2026-02-22T23:23:59.224 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-22T23:23:59.237 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-22T23:23:59.313 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64402 branch: wip-yuri6-testing-2026-02-22-1629-squid description: rados/singleton/{all/ec-inconsistent-hinfo mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} email: yweinste@redhat.com first_in_suite: false flavor: default job_id: '64402' kernel: branch: distro kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: trial name: yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri6-testing-2026-02-22-1629-squid ceph: conf: global: mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd mclock skip benchmark: true osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri6-testing-2026-02-22-1629-squid sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 seed: 5907 sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri6-testing-2026-02-22-1629-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 targets: trial115.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKcSulXW+clEv4dzDKLT66gn41AhCyV7P0t88DikHKhFHCNG6AZtHU3YqyTLqj9M7wqZX8mvVaWcU3WYT27puKQ= tasks: - install: null - ceph: conf: osd: osd max pg log entries: 5 osd min pg log entries: 5 create_rbd_pool: false log-ignorelist: - \(OBJECT_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(SLOW_OPS\) - deep-scrub - missing - overall HEALTH_ - repair - slow request - unfound - \(POOL_APP_NOT_ENABLED\) - enough copies available pre-mgr-commands: - sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force - ec_inconsistent_hinfo: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-22_19:35:40 tube: trial user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653 2026-02-22T23:23:59.313 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa; will attempt to use it 2026-02-22T23:23:59.314 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks 2026-02-22T23:23:59.314 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-22T23:23:59.315 INFO:teuthology.task.internal:Checking packages... 2026-02-22T23:23:59.316 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'f90f4eaebb28b619cf76e80cbbe997656da69579' 2026-02-22T23:23:59.316 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-22T23:23:59.316 INFO:teuthology.packaging:ref: None 2026-02-22T23:23:59.316 INFO:teuthology.packaging:tag: None 2026-02-22T23:23:59.316 INFO:teuthology.packaging:branch: wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T23:23:59.316 INFO:teuthology.packaging:sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T23:23:59.316 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T23:23:59.434 INFO:teuthology.task.internal:Found packages for ceph version 19.2.3-689-gf90f4eae-1jammy 2026-02-22T23:23:59.436 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-22T23:23:59.447 INFO:teuthology.task.internal:no buildpackages task found 2026-02-22T23:23:59.447 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-22T23:23:59.461 INFO:teuthology.task.internal:Saving configuration 2026-02-22T23:23:59.471 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-22T23:23:59.484 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-22T23:23:59.562 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial115.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64402', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-22 23:20:28.231178', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJWg9YRIPxjcwh9gd3sdB4+q6lAhLhVQhVs67VPEz0XW'} 2026-02-22T23:23:59.562 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-22T23:23:59.574 INFO:teuthology.task.internal:roles: ubuntu@trial115.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2026-02-22T23:23:59.574 INFO:teuthology.run_tasks:Running task console_log... 2026-02-22T23:23:59.660 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x742235ef5c60>, signals=[15]) 2026-02-22T23:23:59.660 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-22T23:23:59.675 INFO:teuthology.task.internal:Opening connections... 2026-02-22T23:23:59.675 DEBUG:teuthology.task.internal:connecting to ubuntu@trial115.front.sepia.ceph.com 2026-02-22T23:23:59.676 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T23:23:59.738 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-22T23:23:59.751 DEBUG:teuthology.orchestra.run.trial115:> uname -m 2026-02-22T23:23:59.817 INFO:teuthology.orchestra.run.trial115.stdout:x86_64 2026-02-22T23:23:59.817 DEBUG:teuthology.orchestra.run.trial115:> cat /etc/os-release 2026-02-22T23:23:59.866 INFO:teuthology.orchestra.run.trial115.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-22T23:23:59.867 INFO:teuthology.orchestra.run.trial115.stdout:NAME="Ubuntu" 2026-02-22T23:23:59.867 INFO:teuthology.orchestra.run.trial115.stdout:VERSION_ID="22.04" 2026-02-22T23:23:59.867 INFO:teuthology.orchestra.run.trial115.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-22T23:23:59.867 INFO:teuthology.orchestra.run.trial115.stdout:VERSION_CODENAME=jammy 2026-02-22T23:23:59.867 INFO:teuthology.orchestra.run.trial115.stdout:ID=ubuntu 2026-02-22T23:23:59.867 INFO:teuthology.orchestra.run.trial115.stdout:ID_LIKE=debian 2026-02-22T23:23:59.867 INFO:teuthology.orchestra.run.trial115.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-22T23:23:59.867 INFO:teuthology.orchestra.run.trial115.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-22T23:23:59.867 INFO:teuthology.orchestra.run.trial115.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-22T23:23:59.867 INFO:teuthology.orchestra.run.trial115.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-22T23:23:59.867 INFO:teuthology.orchestra.run.trial115.stdout:UBUNTU_CODENAME=jammy 2026-02-22T23:23:59.868 INFO:teuthology.lock.ops:Updating trial115.front.sepia.ceph.com on lock server 2026-02-22T23:23:59.956 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-22T23:23:59.969 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-22T23:23:59.981 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-22T23:23:59.981 DEBUG:teuthology.orchestra.run.trial115:> test '!' -e /home/ubuntu/cephtest 2026-02-22T23:23:59.984 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-22T23:23:59.994 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-22T23:23:59.994 DEBUG:teuthology.orchestra.run.trial115:> test -z $(ls -A /var/lib/ceph) 2026-02-22T23:24:00.035 INFO:teuthology.orchestra.run.trial115.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-22T23:24:00.036 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-22T23:24:00.135 INFO:teuthology.run_tasks:Running task kernel... 2026-02-22T23:24:00.154 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-22T23:24:00.154 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-22T23:24:00.154 DEBUG:teuthology.orchestra.run.trial115:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-22T23:24:00.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T23:24:00.157 DEBUG:teuthology.orchestra.run.trial115:> uname -r 2026-02-22T23:24:00.202 INFO:teuthology.orchestra.run.trial115.stdout:5.15.0-170-generic 2026-02-22T23:24:00.203 INFO:teuthology.task.kernel:Running kernel on trial115: 5.15.0-170-generic 2026-02-22T23:24:00.203 DEBUG:teuthology.orchestra.run.trial115:> sudo apt-get clean 2026-02-22T23:24:00.260 DEBUG:teuthology.orchestra.run.trial115:> sudo apt-get update 2026-02-22T23:24:00.756 INFO:teuthology.orchestra.run.trial115.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-22T23:24:00.846 INFO:teuthology.orchestra.run.trial115.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-22T23:24:01.183 INFO:teuthology.orchestra.run.trial115.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-22T23:24:01.333 INFO:teuthology.orchestra.run.trial115.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-22T23:24:01.473 INFO:teuthology.orchestra.run.trial115.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3245 kB] 2026-02-22T23:24:01.875 INFO:teuthology.orchestra.run.trial115.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [495 kB] 2026-02-22T23:24:01.886 INFO:teuthology.orchestra.run.trial115.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.1 kB] 2026-02-22T23:24:01.887 INFO:teuthology.orchestra.run.trial115.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-22T23:24:01.922 INFO:teuthology.orchestra.run.trial115.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-22T23:24:01.929 INFO:teuthology.orchestra.run.trial115.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.3 kB] 2026-02-22T23:24:01.930 INFO:teuthology.orchestra.run.trial115.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5310 kB] 2026-02-22T23:24:02.087 INFO:teuthology.orchestra.run.trial115.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1005 kB] 2026-02-22T23:24:02.109 INFO:teuthology.orchestra.run.trial115.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2983 kB] 2026-02-22T23:24:02.196 INFO:teuthology.orchestra.run.trial115.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [427 kB] 2026-02-22T23:24:02.206 INFO:teuthology.orchestra.run.trial115.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-22T23:24:02.206 INFO:teuthology.orchestra.run.trial115.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1018 kB] 2026-02-22T23:24:02.240 INFO:teuthology.orchestra.run.trial115.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.7 kB] 2026-02-22T23:24:02.240 INFO:teuthology.orchestra.run.trial115.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5147 kB] 2026-02-22T23:24:02.384 INFO:teuthology.orchestra.run.trial115.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [979 kB] 2026-02-22T23:24:05.828 INFO:teuthology.orchestra.run.trial115.stdout:Fetched 22.6 MB in 3s (7833 kB/s) 2026-02-22T23:24:06.347 INFO:teuthology.orchestra.run.trial115.stdout:Reading package lists... 2026-02-22T23:24:06.361 DEBUG:teuthology.orchestra.run.trial115:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-22T23:24:06.394 INFO:teuthology.orchestra.run.trial115.stdout:Reading package lists... 2026-02-22T23:24:06.500 INFO:teuthology.orchestra.run.trial115.stdout:Building dependency tree... 2026-02-22T23:24:06.500 INFO:teuthology.orchestra.run.trial115.stdout:Reading state information... 2026-02-22T23:24:06.585 INFO:teuthology.orchestra.run.trial115.stdout:linux-image-generic is already the newest version (5.15.0.170.159). 2026-02-22T23:24:06.585 INFO:teuthology.orchestra.run.trial115.stdout:linux-image-generic set to manually installed. 2026-02-22T23:24:06.585 INFO:teuthology.orchestra.run.trial115.stdout:The following packages were automatically installed and are no longer required: 2026-02-22T23:24:06.585 INFO:teuthology.orchestra.run.trial115.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-22T23:24:06.586 INFO:teuthology.orchestra.run.trial115.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-22T23:24:06.586 INFO:teuthology.orchestra.run.trial115.stdout: sg3-utils-udev 2026-02-22T23:24:06.586 INFO:teuthology.orchestra.run.trial115.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-22T23:24:06.605 INFO:teuthology.orchestra.run.trial115.stdout:0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded. 2026-02-22T23:24:06.606 DEBUG:teuthology.orchestra.run.trial115:> dpkg -s linux-image-generic 2026-02-22T23:24:06.656 INFO:teuthology.orchestra.run.trial115.stdout:Package: linux-image-generic 2026-02-22T23:24:06.657 INFO:teuthology.orchestra.run.trial115.stdout:Status: install ok installed 2026-02-22T23:24:06.657 INFO:teuthology.orchestra.run.trial115.stdout:Priority: optional 2026-02-22T23:24:06.657 INFO:teuthology.orchestra.run.trial115.stdout:Section: kernel 2026-02-22T23:24:06.657 INFO:teuthology.orchestra.run.trial115.stdout:Installed-Size: 21 2026-02-22T23:24:06.657 INFO:teuthology.orchestra.run.trial115.stdout:Maintainer: Ubuntu Kernel Team 2026-02-22T23:24:06.657 INFO:teuthology.orchestra.run.trial115.stdout:Architecture: amd64 2026-02-22T23:24:06.657 INFO:teuthology.orchestra.run.trial115.stdout:Source: linux-meta 2026-02-22T23:24:06.657 INFO:teuthology.orchestra.run.trial115.stdout:Version: 5.15.0.170.159 2026-02-22T23:24:06.657 INFO:teuthology.orchestra.run.trial115.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-22T23:24:06.657 INFO:teuthology.orchestra.run.trial115.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-22T23:24:06.657 INFO:teuthology.orchestra.run.trial115.stdout:Recommends: thermald 2026-02-22T23:24:06.657 INFO:teuthology.orchestra.run.trial115.stdout:Description: Generic Linux kernel image 2026-02-22T23:24:06.657 INFO:teuthology.orchestra.run.trial115.stdout: This package will always depend on the latest generic kernel image 2026-02-22T23:24:06.657 INFO:teuthology.orchestra.run.trial115.stdout: available. 2026-02-22T23:24:06.657 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-22T23:24:06.657 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-22T23:24:06.657 DEBUG:teuthology.orchestra.run.trial115:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-22T23:24:06.707 INFO:teuthology.orchestra.run.trial115.stdout:ttyS1 2026-02-22T23:24:06.741 DEBUG:teuthology.parallel:result is None 2026-02-22T23:24:06.741 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-22T23:24:06.753 INFO:teuthology.task.internal:Creating test directory... 2026-02-22T23:24:06.754 DEBUG:teuthology.orchestra.run.trial115:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-22T23:24:06.787 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-22T23:24:06.799 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-22T23:24:06.813 INFO:teuthology.task.internal:Creating archive directory... 2026-02-22T23:24:06.813 DEBUG:teuthology.orchestra.run.trial115:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-22T23:24:06.832 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-22T23:24:06.846 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-22T23:24:06.846 DEBUG:teuthology.orchestra.run.trial115:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-22T23:24:06.879 INFO:teuthology.orchestra.run.trial115.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:24:06.886 INFO:teuthology.orchestra.run.trial115.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:24:06.887 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-22T23:24:06.900 INFO:teuthology.task.internal:Configuring sudo... 2026-02-22T23:24:06.900 DEBUG:teuthology.orchestra.run.trial115:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-22T23:24:06.935 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-22T23:24:06.948 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-22T23:24:06.948 DEBUG:teuthology.orchestra.run.trial115:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-22T23:24:06.979 DEBUG:teuthology.orchestra.run.trial115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T23:24:07.023 DEBUG:teuthology.orchestra.run.trial115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T23:24:07.071 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:24:07.072 DEBUG:teuthology.orchestra.run.trial115:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-22T23:24:07.125 DEBUG:teuthology.orchestra.run.trial115:> sudo service rsyslog restart 2026-02-22T23:24:07.185 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-22T23:24:07.200 INFO:teuthology.task.internal:Starting timer... 2026-02-22T23:24:07.200 INFO:teuthology.run_tasks:Running task pcp... 2026-02-22T23:24:07.218 INFO:teuthology.run_tasks:Running task selinux... 2026-02-22T23:24:07.235 DEBUG:teuthology.task.selinux:Excluding trial115: OS 'ubuntu' does not support SELinux 2026-02-22T23:24:07.235 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-22T23:24:07.235 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-22T23:24:07.236 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-22T23:24:07.236 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-22T23:24:07.252 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-22T23:24:07.264 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-22T23:24:07.265 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial115.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-22T23:26:29.031 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial115.front.sepia.ceph.com')] 2026-02-22T23:26:29.032 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial115.front.sepia.ceph.com' 2026-02-22T23:26:29.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T23:26:29.122 DEBUG:teuthology.orchestra.run.trial115:> true 2026-02-22T23:26:29.369 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial115.front.sepia.ceph.com' 2026-02-22T23:26:29.370 INFO:teuthology.run_tasks:Running task clock... 2026-02-22T23:26:29.384 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-22T23:26:29.384 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-22T23:26:29.385 DEBUG:teuthology.orchestra.run.trial115:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T23:26:29.429 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-22T23:26:29.429 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: Command line: ntpd -gq 2026-02-22T23:26:29.429 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: ---------------------------------------------------- 2026-02-22T23:26:29.429 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: ntp-4 is maintained by Network Time Foundation, 2026-02-22T23:26:29.429 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-22T23:26:29.429 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: corporation. Support and training for ntp-4 are 2026-02-22T23:26:29.429 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: available at https://www.nwtime.org/support 2026-02-22T23:26:29.429 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: ---------------------------------------------------- 2026-02-22T23:26:29.429 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: proto: precision = 0.020 usec (-25) 2026-02-22T23:26:29.430 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: basedate set to 2022-02-04 2026-02-22T23:26:29.430 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: gps base set to 2022-02-06 (week 2196) 2026-02-22T23:26:29.430 INFO:teuthology.orchestra.run.trial115.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-22T23:26:29.430 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-22T23:26:29.430 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: restrict ::: KOD does nothing without LIMITED. 2026-02-22T23:26:29.430 INFO:teuthology.orchestra.run.trial115.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-22T23:26:29.431 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: Listen and drop on 0 v6wildcard [::]:123 2026-02-22T23:26:29.431 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-22T23:26:29.431 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: Listen normally on 2 lo 127.0.0.1:123 2026-02-22T23:26:29.431 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: Listen normally on 3 enp1s0f0 10.20.193.115:123 2026-02-22T23:26:29.431 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: Listen normally on 4 lo [::1]:123 2026-02-22T23:26:29.431 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:6154%2]:123 2026-02-22T23:26:29.431 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:29 ntpd[9419]: Listening on routing socket on fd #22 for interface updates 2026-02-22T23:26:30.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:30 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:30.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:30 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:30.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:30 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:30.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:30 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:30.433 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:30 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:30.433 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:30 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:30.433 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:30 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T23:26:30.433 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:30 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T23:26:30.527 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:30 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:30.528 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:30 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:30.528 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:30 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T23:26:30.528 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:30 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T23:26:31.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:31 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:31.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:31 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:31.497 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:31 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:31.497 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:31 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:31.497 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:31 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T23:26:31.497 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:31 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T23:26:32.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:32 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:32.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:32 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:32.430 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:32 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:32.430 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:32 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:32.430 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:32 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:32.430 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:32 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:32.433 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:32 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:32.433 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:32 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:32.433 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:32 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T23:26:32.433 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:32 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T23:26:32.456 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:32 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:32.456 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:32 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:32.456 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:32 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T23:26:32.456 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:32 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T23:26:32.527 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:32 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:32.527 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:32 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:32.527 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:32 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T23:26:32.527 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:32 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T23:26:33.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:33 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:33.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:33 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:33.497 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:33 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:33.497 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:33 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:33.497 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:33 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T23:26:33.497 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:33 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T23:26:34.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:34 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:34.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:34 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:34.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:34 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:34.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:34 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:34.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:34 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:34.430 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:34 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:34.433 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:34 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:34.433 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:34 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:34.433 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:34 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T23:26:34.433 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:34 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T23:26:34.457 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:34 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:34.457 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:34 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:34.458 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:34 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T23:26:34.458 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:34 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T23:26:34.528 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:34 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:34.528 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:34 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:34.528 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:34 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T23:26:34.528 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:34 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T23:26:35.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:35 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:35.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:35 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:35.497 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:35 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:35.498 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:35 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:35.498 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:35 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T23:26:35.498 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:35 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T23:26:36.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:36 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:36.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:36 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:36.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:36 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:36.429 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:36 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:36.430 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:36 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:36.430 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:36 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:36.433 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:36 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T23:26:36.433 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:36 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T23:26:36.433 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:36 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T23:26:36.433 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:36 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T23:26:36.433 INFO:teuthology.orchestra.run.trial115.stdout:22 Feb 23:26:36 ntpd[9419]: ntpd: time set -0.441221 s 2026-02-22T23:26:36.433 INFO:teuthology.orchestra.run.trial115.stdout:ntpd: time set -0.441221s 2026-02-22T23:26:36.434 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:36 ntpd[9419]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-22T23:26:36.434 INFO:teuthology.orchestra.run.trial115.stderr:22 Feb 23:26:36 ntpd[9419]: can't open /var/log/ntpstats/loopstats.20260222: Permission denied 2026-02-22T23:26:36.458 INFO:teuthology.orchestra.run.trial115.stdout:No association ID's returned 2026-02-22T23:26:36.459 INFO:teuthology.run_tasks:Running task install... 2026-02-22T23:26:36.473 DEBUG:teuthology.task.install:project ceph 2026-02-22T23:26:36.473 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}} 2026-02-22T23:26:36.473 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'} 2026-02-22T23:26:36.473 INFO:teuthology.task.install:Using flavor: default 2026-02-22T23:26:36.482 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-22T23:26:36.482 INFO:teuthology.task.install:extra packages: [] 2026-02-22T23:26:36.483 DEBUG:teuthology.orchestra.run.trial115:> sudo apt-key list | grep Ceph 2026-02-22T23:26:36.551 INFO:teuthology.orchestra.run.trial115.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-22T23:26:36.579 INFO:teuthology.orchestra.run.trial115.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-22T23:26:36.579 INFO:teuthology.orchestra.run.trial115.stdout:uid [ unknown] Ceph.com (release key) 2026-02-22T23:26:36.580 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2026-02-22T23:26:36.581 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T23:26:36.748 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default/ 2026-02-22T23:26:36.748 INFO:teuthology.task.install.deb:Package version is 19.2.3-689-gf90f4eae-1jammy 2026-02-22T23:26:36.889 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:26:36.889 DEBUG:teuthology.orchestra.run.trial115:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-22T23:26:36.899 DEBUG:teuthology.orchestra.run.trial115:> sudo apt-get update 2026-02-22T23:26:37.070 INFO:teuthology.orchestra.run.trial115.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy InRelease 2026-02-22T23:26:37.081 INFO:teuthology.orchestra.run.trial115.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy Release [7662 B] 2026-02-22T23:26:37.097 INFO:teuthology.orchestra.run.trial115.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-22T23:26:37.109 INFO:teuthology.orchestra.run.trial115.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2026-02-22T23:26:37.355 INFO:teuthology.orchestra.run.trial115.stdout:Hit:5 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-22T23:26:37.423 INFO:teuthology.orchestra.run.trial115.stdout:Hit:6 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-22T23:26:37.495 INFO:teuthology.orchestra.run.trial115.stdout:Hit:7 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-22T23:26:37.568 INFO:teuthology.orchestra.run.trial115.stdout:Hit:8 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-22T23:26:37.714 INFO:teuthology.orchestra.run.trial115.stdout:Fetched 25.8 kB in 1s (39.5 kB/s) 2026-02-22T23:26:38.247 INFO:teuthology.orchestra.run.trial115.stdout:Reading package lists... 2026-02-22T23:26:38.261 DEBUG:teuthology.orchestra.run.trial115:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.2.3-689-gf90f4eae-1jammy cephadm=19.2.3-689-gf90f4eae-1jammy ceph-mds=19.2.3-689-gf90f4eae-1jammy ceph-mgr=19.2.3-689-gf90f4eae-1jammy ceph-common=19.2.3-689-gf90f4eae-1jammy ceph-fuse=19.2.3-689-gf90f4eae-1jammy ceph-test=19.2.3-689-gf90f4eae-1jammy ceph-volume=19.2.3-689-gf90f4eae-1jammy radosgw=19.2.3-689-gf90f4eae-1jammy python3-rados=19.2.3-689-gf90f4eae-1jammy python3-rgw=19.2.3-689-gf90f4eae-1jammy python3-cephfs=19.2.3-689-gf90f4eae-1jammy python3-rbd=19.2.3-689-gf90f4eae-1jammy libcephfs2=19.2.3-689-gf90f4eae-1jammy libcephfs-dev=19.2.3-689-gf90f4eae-1jammy librados2=19.2.3-689-gf90f4eae-1jammy librbd1=19.2.3-689-gf90f4eae-1jammy rbd-fuse=19.2.3-689-gf90f4eae-1jammy 2026-02-22T23:26:38.294 INFO:teuthology.orchestra.run.trial115.stdout:Reading package lists... 2026-02-22T23:26:38.407 INFO:teuthology.orchestra.run.trial115.stdout:Building dependency tree... 2026-02-22T23:26:38.408 INFO:teuthology.orchestra.run.trial115.stdout:Reading state information... 2026-02-22T23:26:38.501 INFO:teuthology.orchestra.run.trial115.stdout:The following packages were automatically installed and are no longer required: 2026-02-22T23:26:38.501 INFO:teuthology.orchestra.run.trial115.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-22T23:26:38.501 INFO:teuthology.orchestra.run.trial115.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-22T23:26:38.501 INFO:teuthology.orchestra.run.trial115.stdout: sg3-utils-udev 2026-02-22T23:26:38.501 INFO:teuthology.orchestra.run.trial115.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-22T23:26:38.501 INFO:teuthology.orchestra.run.trial115.stdout:The following additional packages will be installed: 2026-02-22T23:26:38.502 INFO:teuthology.orchestra.run.trial115.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-22T23:26:38.502 INFO:teuthology.orchestra.run.trial115.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2026-02-22T23:26:38.502 INFO:teuthology.orchestra.run.trial115.stdout: libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 2026-02-22T23:26:38.502 INFO:teuthology.orchestra.run.trial115.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-22T23:26:38.502 INFO:teuthology.orchestra.run.trial115.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2026-02-22T23:26:38.502 INFO:teuthology.orchestra.run.trial115.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2026-02-22T23:26:38.502 INFO:teuthology.orchestra.run.trial115.stdout: pkg-config python-asyncssh-doc python-pastedeploy-tpl python3-asyncssh 2026-02-22T23:26:38.502 INFO:teuthology.orchestra.run.trial115.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cheroot 2026-02-22T23:26:38.502 INFO:teuthology.orchestra.run.trial115.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-22T23:26:38.502 INFO:teuthology.orchestra.run.trial115.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-22T23:26:38.502 INFO:teuthology.orchestra.run.trial115.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-22T23:26:38.502 INFO:teuthology.orchestra.run.trial115.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-22T23:26:38.503 INFO:teuthology.orchestra.run.trial115.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-22T23:26:38.503 INFO:teuthology.orchestra.run.trial115.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-22T23:26:38.503 INFO:teuthology.orchestra.run.trial115.stdout: python3-pyinotify python3-pytest python3-repoze.lru 2026-02-22T23:26:38.503 INFO:teuthology.orchestra.run.trial115.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2026-02-22T23:26:38.503 INFO:teuthology.orchestra.run.trial115.stdout: python3-simplejson python3-singledispatch python3-sklearn 2026-02-22T23:26:38.503 INFO:teuthology.orchestra.run.trial115.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2026-02-22T23:26:38.503 INFO:teuthology.orchestra.run.trial115.stdout: python3-toml python3-waitress python3-wcwidth python3-webob 2026-02-22T23:26:38.503 INFO:teuthology.orchestra.run.trial115.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2026-02-22T23:26:38.503 INFO:teuthology.orchestra.run.trial115.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2026-02-22T23:26:38.503 INFO:teuthology.orchestra.run.trial115.stdout:Suggested packages: 2026-02-22T23:26:38.503 INFO:teuthology.orchestra.run.trial115.stdout: python3-influxdb readline-doc python3-beaker python-mako-doc 2026-02-22T23:26:38.503 INFO:teuthology.orchestra.run.trial115.stdout: python-natsort-doc httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2026-02-22T23:26:38.504 INFO:teuthology.orchestra.run.trial115.stdout: libjs-mochikit python-pecan-doc python-psutil-doc subversion 2026-02-22T23:26:38.504 INFO:teuthology.orchestra.run.trial115.stdout: python-pygments-doc ttf-bitstream-vera python-pyinotify-doc python3-dap 2026-02-22T23:26:38.504 INFO:teuthology.orchestra.run.trial115.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2026-02-22T23:26:38.504 INFO:teuthology.orchestra.run.trial115.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2026-02-22T23:26:38.504 INFO:teuthology.orchestra.run.trial115.stdout:Recommended packages: 2026-02-22T23:26:38.504 INFO:teuthology.orchestra.run.trial115.stdout: btrfs-tools 2026-02-22T23:26:38.535 INFO:teuthology.orchestra.run.trial115.stdout:The following NEW packages will be installed: 2026-02-22T23:26:38.535 INFO:teuthology.orchestra.run.trial115.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-22T23:26:38.535 INFO:teuthology.orchestra.run.trial115.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-22T23:26:38.535 INFO:teuthology.orchestra.run.trial115.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2026-02-22T23:26:38.536 INFO:teuthology.orchestra.run.trial115.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 2026-02-22T23:26:38.536 INFO:teuthology.orchestra.run.trial115.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2026-02-22T23:26:38.536 INFO:teuthology.orchestra.run.trial115.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2026-02-22T23:26:38.536 INFO:teuthology.orchestra.run.trial115.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2026-02-22T23:26:38.537 INFO:teuthology.orchestra.run.trial115.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2026-02-22T23:26:38.537 INFO:teuthology.orchestra.run.trial115.stdout: python-pastedeploy-tpl python3-asyncssh python3-cachetools 2026-02-22T23:26:38.537 INFO:teuthology.orchestra.run.trial115.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cheroot 2026-02-22T23:26:38.537 INFO:teuthology.orchestra.run.trial115.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-22T23:26:38.537 INFO:teuthology.orchestra.run.trial115.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-22T23:26:38.538 INFO:teuthology.orchestra.run.trial115.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-22T23:26:38.538 INFO:teuthology.orchestra.run.trial115.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-22T23:26:38.538 INFO:teuthology.orchestra.run.trial115.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-22T23:26:38.538 INFO:teuthology.orchestra.run.trial115.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-22T23:26:38.538 INFO:teuthology.orchestra.run.trial115.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2026-02-22T23:26:38.538 INFO:teuthology.orchestra.run.trial115.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2026-02-22T23:26:38.538 INFO:teuthology.orchestra.run.trial115.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2026-02-22T23:26:38.538 INFO:teuthology.orchestra.run.trial115.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2026-02-22T23:26:38.538 INFO:teuthology.orchestra.run.trial115.stdout: python3-threadpoolctl python3-toml python3-waitress python3-wcwidth 2026-02-22T23:26:38.538 INFO:teuthology.orchestra.run.trial115.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2026-02-22T23:26:38.538 INFO:teuthology.orchestra.run.trial115.stdout: python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse socat unzip 2026-02-22T23:26:38.538 INFO:teuthology.orchestra.run.trial115.stdout: xmlstarlet zip 2026-02-22T23:26:38.624 INFO:teuthology.orchestra.run.trial115.stdout:0 upgraded, 108 newly installed, 0 to remove and 13 not upgraded. 2026-02-22T23:26:38.624 INFO:teuthology.orchestra.run.trial115.stdout:Need to get 178 MB of archives. 2026-02-22T23:26:38.624 INFO:teuthology.orchestra.run.trial115.stdout:After this operation, 809 MB of additional disk space will be used. 2026-02-22T23:26:38.624 INFO:teuthology.orchestra.run.trial115.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.2.3-689-gf90f4eae-1jammy [3597 kB] 2026-02-22T23:26:38.731 INFO:teuthology.orchestra.run.trial115.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.2.3-689-gf90f4eae-1jammy [3256 kB] 2026-02-22T23:26:38.780 INFO:teuthology.orchestra.run.trial115.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.2.3-689-gf90f4eae-1jammy [979 kB] 2026-02-22T23:26:38.795 INFO:teuthology.orchestra.run.trial115.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.2.3-689-gf90f4eae-1jammy [357 kB] 2026-02-22T23:26:38.800 INFO:teuthology.orchestra.run.trial115.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.2.3-689-gf90f4eae-1jammy [32.9 kB] 2026-02-22T23:26:38.801 INFO:teuthology.orchestra.run.trial115.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.2.3-689-gf90f4eae-1jammy [184 kB] 2026-02-22T23:26:38.804 INFO:teuthology.orchestra.run.trial115.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.2.3-689-gf90f4eae-1jammy [70.0 kB] 2026-02-22T23:26:38.805 INFO:teuthology.orchestra.run.trial115.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.2.3-689-gf90f4eae-1jammy [335 kB] 2026-02-22T23:26:38.813 INFO:teuthology.orchestra.run.trial115.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.2.3-689-gf90f4eae-1jammy [6935 kB] 2026-02-22T23:26:38.890 INFO:teuthology.orchestra.run.trial115.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-22T23:26:38.936 INFO:teuthology.orchestra.run.trial115.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.2.3-689-gf90f4eae-1jammy [112 kB] 2026-02-22T23:26:38.937 INFO:teuthology.orchestra.run.trial115.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.2.3-689-gf90f4eae-1jammy [470 kB] 2026-02-22T23:26:38.945 INFO:teuthology.orchestra.run.trial115.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.2.3-689-gf90f4eae-1jammy [26.5 MB] 2026-02-22T23:26:39.226 INFO:teuthology.orchestra.run.trial115.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-22T23:26:39.237 INFO:teuthology.orchestra.run.trial115.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-22T23:26:39.306 INFO:teuthology.orchestra.run.trial115.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-22T23:26:39.424 INFO:teuthology.orchestra.run.trial115.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.2.3-689-gf90f4eae-1jammy [5178 kB] 2026-02-22T23:26:39.507 INFO:teuthology.orchestra.run.trial115.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-22T23:26:39.517 INFO:teuthology.orchestra.run.trial115.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-22T23:26:39.519 INFO:teuthology.orchestra.run.trial115.stdout:Get:20 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.2.3-689-gf90f4eae-1jammy [248 kB] 2026-02-22T23:26:39.523 INFO:teuthology.orchestra.run.trial115.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.2.3-689-gf90f4eae-1jammy [125 kB] 2026-02-22T23:26:39.525 INFO:teuthology.orchestra.run.trial115.stdout:Get:22 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.2.3-689-gf90f4eae-1jammy [1081 kB] 2026-02-22T23:26:39.543 INFO:teuthology.orchestra.run.trial115.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-22T23:26:39.544 INFO:teuthology.orchestra.run.trial115.stdout:Get:24 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.2.3-689-gf90f4eae-1jammy [6239 kB] 2026-02-22T23:26:39.551 INFO:teuthology.orchestra.run.trial115.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-22T23:26:39.553 INFO:teuthology.orchestra.run.trial115.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-22T23:26:39.554 INFO:teuthology.orchestra.run.trial115.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-22T23:26:39.555 INFO:teuthology.orchestra.run.trial115.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-22T23:26:39.572 INFO:teuthology.orchestra.run.trial115.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2026-02-22T23:26:39.576 INFO:teuthology.orchestra.run.trial115.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2026-02-22T23:26:39.579 INFO:teuthology.orchestra.run.trial115.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2026-02-22T23:26:39.647 INFO:teuthology.orchestra.run.trial115.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5034 B] 2026-02-22T23:26:39.648 INFO:teuthology.orchestra.run.trial115.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2026-02-22T23:26:39.649 INFO:teuthology.orchestra.run.trial115.stdout:Get:34 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2026-02-22T23:26:39.651 INFO:teuthology.orchestra.run.trial115.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2026-02-22T23:26:39.653 INFO:teuthology.orchestra.run.trial115.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-22T23:26:39.653 INFO:teuthology.orchestra.run.trial115.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-22T23:26:39.654 INFO:teuthology.orchestra.run.trial115.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-22T23:26:39.654 INFO:teuthology.orchestra.run.trial115.stdout:Get:39 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-22T23:26:39.660 INFO:teuthology.orchestra.run.trial115.stdout:Get:40 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.2.3-689-gf90f4eae-1jammy [23.0 MB] 2026-02-22T23:26:39.726 INFO:teuthology.orchestra.run.trial115.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-22T23:26:39.726 INFO:teuthology.orchestra.run.trial115.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-22T23:26:39.726 INFO:teuthology.orchestra.run.trial115.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-22T23:26:39.726 INFO:teuthology.orchestra.run.trial115.stdout:Get:44 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-22T23:26:39.727 INFO:teuthology.orchestra.run.trial115.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-22T23:26:39.797 INFO:teuthology.orchestra.run.trial115.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-22T23:26:39.798 INFO:teuthology.orchestra.run.trial115.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-22T23:26:39.799 INFO:teuthology.orchestra.run.trial115.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2026-02-22T23:26:39.799 INFO:teuthology.orchestra.run.trial115.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2026-02-22T23:26:39.800 INFO:teuthology.orchestra.run.trial115.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2026-02-22T23:26:39.868 INFO:teuthology.orchestra.run.trial115.stdout:Get:51 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7320 B] 2026-02-22T23:26:39.869 INFO:teuthology.orchestra.run.trial115.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-22T23:26:39.869 INFO:teuthology.orchestra.run.trial115.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-waitress all 1.4.4-1.1ubuntu1.1 [47.0 kB] 2026-02-22T23:26:39.870 INFO:teuthology.orchestra.run.trial115.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2026-02-22T23:26:39.870 INFO:teuthology.orchestra.run.trial115.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2026-02-22T23:26:39.941 INFO:teuthology.orchestra.run.trial115.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4892 B] 2026-02-22T23:26:39.941 INFO:teuthology.orchestra.run.trial115.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2026-02-22T23:26:39.941 INFO:teuthology.orchestra.run.trial115.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2026-02-22T23:26:39.942 INFO:teuthology.orchestra.run.trial115.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2026-02-22T23:26:39.942 INFO:teuthology.orchestra.run.trial115.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.3 [181 kB] 2026-02-22T23:26:40.013 INFO:teuthology.orchestra.run.trial115.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-22T23:26:40.013 INFO:teuthology.orchestra.run.trial115.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-22T23:26:40.016 INFO:teuthology.orchestra.run.trial115.stdout:Get:63 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-22T23:26:40.016 INFO:teuthology.orchestra.run.trial115.stdout:Get:64 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-22T23:26:40.017 INFO:teuthology.orchestra.run.trial115.stdout:Get:65 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-22T23:26:40.084 INFO:teuthology.orchestra.run.trial115.stdout:Get:66 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-22T23:26:40.086 INFO:teuthology.orchestra.run.trial115.stdout:Get:67 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-22T23:26:40.087 INFO:teuthology.orchestra.run.trial115.stdout:Get:68 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-22T23:26:40.101 INFO:teuthology.orchestra.run.trial115.stdout:Get:69 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.2.3-689-gf90f4eae-1jammy [14.2 kB] 2026-02-22T23:26:40.101 INFO:teuthology.orchestra.run.trial115.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.2.3-689-gf90f4eae-1jammy [1173 kB] 2026-02-22T23:26:40.107 INFO:teuthology.orchestra.run.trial115.stdout:Get:71 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-22T23:26:40.107 INFO:teuthology.orchestra.run.trial115.stdout:Get:72 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-22T23:26:40.118 INFO:teuthology.orchestra.run.trial115.stdout:Get:73 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.2.3-689-gf90f4eae-1jammy [2503 kB] 2026-02-22T23:26:40.156 INFO:teuthology.orchestra.run.trial115.stdout:Get:74 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-22T23:26:40.157 INFO:teuthology.orchestra.run.trial115.stdout:Get:75 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-22T23:26:40.157 INFO:teuthology.orchestra.run.trial115.stdout:Get:76 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-22T23:26:40.157 INFO:teuthology.orchestra.run.trial115.stdout:Get:77 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-22T23:26:40.160 INFO:teuthology.orchestra.run.trial115.stdout:Get:78 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-22T23:26:40.169 INFO:teuthology.orchestra.run.trial115.stdout:Get:79 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.2.3-689-gf90f4eae-1jammy [798 kB] 2026-02-22T23:26:40.182 INFO:teuthology.orchestra.run.trial115.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.2.3-689-gf90f4eae-1jammy [157 kB] 2026-02-22T23:26:40.184 INFO:teuthology.orchestra.run.trial115.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.2.3-689-gf90f4eae-1jammy [2396 kB] 2026-02-22T23:26:40.228 INFO:teuthology.orchestra.run.trial115.stdout:Get:82 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-22T23:26:40.230 INFO:teuthology.orchestra.run.trial115.stdout:Get:83 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-22T23:26:40.230 INFO:teuthology.orchestra.run.trial115.stdout:Get:84 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-22T23:26:40.233 INFO:teuthology.orchestra.run.trial115.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.2.3-689-gf90f4eae-1jammy [8625 kB] 2026-02-22T23:26:40.234 INFO:teuthology.orchestra.run.trial115.stdout:Get:86 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-22T23:26:40.236 INFO:teuthology.orchestra.run.trial115.stdout:Get:87 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2026-02-22T23:26:40.300 INFO:teuthology.orchestra.run.trial115.stdout:Get:88 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2026-02-22T23:26:40.301 INFO:teuthology.orchestra.run.trial115.stdout:Get:89 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-22T23:26:40.305 INFO:teuthology.orchestra.run.trial115.stdout:Get:90 https://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2026-02-22T23:26:40.306 INFO:teuthology.orchestra.run.trial115.stdout:Get:91 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-22T23:26:40.308 INFO:teuthology.orchestra.run.trial115.stdout:Get:92 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-22T23:26:40.372 INFO:teuthology.orchestra.run.trial115.stdout:Get:93 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2026-02-22T23:26:40.373 INFO:teuthology.orchestra.run.trial115.stdout:Get:94 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-22T23:26:40.373 INFO:teuthology.orchestra.run.trial115.stdout:Get:95 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-22T23:26:40.374 INFO:teuthology.orchestra.run.trial115.stdout:Get:96 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-22T23:26:40.375 INFO:teuthology.orchestra.run.trial115.stdout:Get:97 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-22T23:26:40.407 INFO:teuthology.orchestra.run.trial115.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.2.3-689-gf90f4eae-1jammy [14.3 kB] 2026-02-22T23:26:40.407 INFO:teuthology.orchestra.run.trial115.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.2.3-689-gf90f4eae-1jammy [52.1 MB] 2026-02-22T23:26:40.444 INFO:teuthology.orchestra.run.trial115.stdout:Get:100 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2026-02-22T23:26:40.445 INFO:teuthology.orchestra.run.trial115.stdout:Get:101 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-22T23:26:40.445 INFO:teuthology.orchestra.run.trial115.stdout:Get:102 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-22T23:26:40.447 INFO:teuthology.orchestra.run.trial115.stdout:Get:103 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-22T23:26:40.447 INFO:teuthology.orchestra.run.trial115.stdout:Get:104 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-22T23:26:41.858 INFO:teuthology.orchestra.run.trial115.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.2.3-689-gf90f4eae-1jammy [135 kB] 2026-02-22T23:26:41.859 INFO:teuthology.orchestra.run.trial115.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.2.3-689-gf90f4eae-1jammy [41.0 kB] 2026-02-22T23:26:41.859 INFO:teuthology.orchestra.run.trial115.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.2.3-689-gf90f4eae-1jammy [13.7 MB] 2026-02-22T23:26:42.151 INFO:teuthology.orchestra.run.trial115.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.2.3-689-gf90f4eae-1jammy [92.3 kB] 2026-02-22T23:26:42.469 INFO:teuthology.orchestra.run.trial115.stdout:Fetched 178 MB in 4s (49.2 MB/s) 2026-02-22T23:26:42.486 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-22T23:26:42.642 INFO:teuthology.orchestra.run.trial115.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-22T23:26:42.644 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-22T23:26:42.645 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-22T23:26:42.667 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-22T23:26:42.676 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-22T23:26:42.676 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-22T23:26:42.692 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-22T23:26:42.704 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-22T23:26:42.705 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-22T23:26:42.723 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-22T23:26:42.731 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T23:26:42.736 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T23:26:42.765 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-22T23:26:42.773 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T23:26:42.773 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T23:26:42.789 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-22T23:26:42.797 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T23:26:42.797 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T23:26:42.817 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-22T23:26:42.825 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-22T23:26:42.825 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-22T23:26:42.841 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package librados2. 2026-02-22T23:26:42.849 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../007-librados2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:42.849 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking librados2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:42.889 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package libnbd0. 2026-02-22T23:26:42.898 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2026-02-22T23:26:42.898 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-22T23:26:42.912 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package librbd1. 2026-02-22T23:26:42.920 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../009-librbd1_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:42.920 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking librbd1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:42.963 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package libcephfs2. 2026-02-22T23:26:42.971 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../010-libcephfs2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:42.971 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking libcephfs2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:42.991 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-rados. 2026-02-22T23:26:42.999 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../011-python3-rados_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:42.999 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-rados (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:43.017 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-22T23:26:43.025 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T23:26:43.025 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-ceph-argparse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:43.039 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-cephfs. 2026-02-22T23:26:43.047 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../013-python3-cephfs_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:43.047 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-cephfs (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:43.064 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-22T23:26:43.072 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../014-python3-ceph-common_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T23:26:43.073 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:43.090 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-22T23:26:43.098 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-22T23:26:43.098 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-22T23:26:43.113 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-prettytable. 2026-02-22T23:26:43.121 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2026-02-22T23:26:43.122 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-22T23:26:43.136 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-rbd. 2026-02-22T23:26:43.144 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../017-python3-rbd_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:43.144 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-rbd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:43.162 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-22T23:26:43.170 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-22T23:26:43.170 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-22T23:26:43.188 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package libreadline-dev:amd64. 2026-02-22T23:26:43.196 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2026-02-22T23:26:43.196 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2026-02-22T23:26:43.214 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2026-02-22T23:26:43.222 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2026-02-22T23:26:43.223 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-22T23:26:43.239 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package lua5.1. 2026-02-22T23:26:43.247 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2026-02-22T23:26:43.248 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2026-02-22T23:26:43.267 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package lua-any. 2026-02-22T23:26:43.274 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2026-02-22T23:26:43.275 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking lua-any (27ubuntu1) ... 2026-02-22T23:26:43.288 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package zip. 2026-02-22T23:26:43.296 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2026-02-22T23:26:43.297 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking zip (3.0-12build2) ... 2026-02-22T23:26:43.313 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package unzip. 2026-02-22T23:26:43.321 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2026-02-22T23:26:43.321 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2026-02-22T23:26:43.339 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package luarocks. 2026-02-22T23:26:43.347 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2026-02-22T23:26:43.347 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2026-02-22T23:26:43.385 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package librgw2. 2026-02-22T23:26:43.394 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../026-librgw2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:43.394 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking librgw2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:43.466 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-rgw. 2026-02-22T23:26:43.474 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../027-python3-rgw_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:43.474 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-rgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:43.489 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-22T23:26:43.497 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../028-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-22T23:26:43.498 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-22T23:26:43.513 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package libradosstriper1. 2026-02-22T23:26:43.521 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../029-libradosstriper1_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:43.521 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking libradosstriper1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:43.540 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package ceph-common. 2026-02-22T23:26:43.548 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../030-ceph-common_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:43.548 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:43.749 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package ceph-base. 2026-02-22T23:26:43.758 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../031-ceph-base_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:43.762 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking ceph-base (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:43.826 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-22T23:26:43.834 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-22T23:26:43.834 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-22T23:26:43.848 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-cheroot. 2026-02-22T23:26:43.856 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-22T23:26:43.857 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-22T23:26:43.873 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-22T23:26:43.882 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-22T23:26:43.882 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-22T23:26:43.897 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-22T23:26:43.905 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-22T23:26:43.905 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-22T23:26:43.920 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-22T23:26:43.928 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-22T23:26:43.929 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-22T23:26:43.943 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-tempora. 2026-02-22T23:26:43.951 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2026-02-22T23:26:43.952 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-22T23:26:43.967 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-portend. 2026-02-22T23:26:43.975 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2026-02-22T23:26:43.976 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-22T23:26:43.990 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-22T23:26:43.999 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-22T23:26:43.999 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-22T23:26:44.014 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-22T23:26:44.022 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-22T23:26:44.022 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-22T23:26:44.045 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-natsort. 2026-02-22T23:26:44.053 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2026-02-22T23:26:44.054 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-22T23:26:44.070 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-logutils. 2026-02-22T23:26:44.078 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2026-02-22T23:26:44.078 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-logutils (0.3.3-8) ... 2026-02-22T23:26:44.093 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-mako. 2026-02-22T23:26:44.101 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2026-02-22T23:26:44.101 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-22T23:26:44.119 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-simplegeneric. 2026-02-22T23:26:44.127 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2026-02-22T23:26:44.127 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2026-02-22T23:26:44.142 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-singledispatch. 2026-02-22T23:26:44.151 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2026-02-22T23:26:44.151 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2026-02-22T23:26:44.164 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-webob. 2026-02-22T23:26:44.172 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-22T23:26:44.173 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-22T23:26:44.189 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-waitress. 2026-02-22T23:26:44.197 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1.1_all.deb ... 2026-02-22T23:26:44.199 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-22T23:26:44.215 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-tempita. 2026-02-22T23:26:44.223 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2026-02-22T23:26:44.223 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2026-02-22T23:26:44.238 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-paste. 2026-02-22T23:26:44.245 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2026-02-22T23:26:44.246 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2026-02-22T23:26:44.274 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2026-02-22T23:26:44.283 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2026-02-22T23:26:44.283 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2026-02-22T23:26:44.297 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-pastedeploy. 2026-02-22T23:26:44.306 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2026-02-22T23:26:44.306 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2026-02-22T23:26:44.322 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-webtest. 2026-02-22T23:26:44.330 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2026-02-22T23:26:44.330 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-webtest (2.0.35-1) ... 2026-02-22T23:26:44.345 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-pecan. 2026-02-22T23:26:44.353 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2026-02-22T23:26:44.353 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2026-02-22T23:26:44.379 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-werkzeug. 2026-02-22T23:26:44.386 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.3_all.deb ... 2026-02-22T23:26:44.386 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-22T23:26:44.406 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-22T23:26:44.415 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T23:26:44.415 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking ceph-mgr-modules-core (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:44.443 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-22T23:26:44.451 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:44.452 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking libsqlite3-mod-ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:44.467 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package ceph-mgr. 2026-02-22T23:26:44.475 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../057-ceph-mgr_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:44.475 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking ceph-mgr (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:44.497 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package ceph-mon. 2026-02-22T23:26:44.505 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../058-ceph-mon_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:44.506 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking ceph-mon (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:44.564 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-22T23:26:44.572 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../059-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-22T23:26:44.573 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-22T23:26:44.589 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package ceph-osd. 2026-02-22T23:26:44.597 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../060-ceph-osd_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:44.597 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking ceph-osd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:44.805 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package ceph. 2026-02-22T23:26:44.814 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../061-ceph_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:44.814 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:44.828 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package ceph-fuse. 2026-02-22T23:26:44.836 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../062-ceph-fuse_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:44.836 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking ceph-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:44.860 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package ceph-mds. 2026-02-22T23:26:44.868 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../063-ceph-mds_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:44.869 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking ceph-mds (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:44.903 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package cephadm. 2026-02-22T23:26:44.911 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../064-cephadm_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:44.912 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:44.928 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-22T23:26:44.936 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-22T23:26:44.937 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-22T23:26:44.958 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-22T23:26:44.966 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../066-ceph-mgr-cephadm_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T23:26:44.967 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking ceph-mgr-cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:44.987 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-22T23:26:44.994 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../067-python3-repoze.lru_0.7-2_all.deb ... 2026-02-22T23:26:44.995 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-22T23:26:45.010 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-routes. 2026-02-22T23:26:45.018 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../068-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-22T23:26:45.019 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-22T23:26:45.038 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-22T23:26:45.046 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../069-ceph-mgr-dashboard_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T23:26:45.047 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking ceph-mgr-dashboard (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:45.250 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-22T23:26:45.259 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../070-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-22T23:26:45.259 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-22T23:26:45.297 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-joblib. 2026-02-22T23:26:45.305 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../071-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-22T23:26:45.305 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-22T23:26:45.332 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-22T23:26:45.340 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../072-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-22T23:26:45.340 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-22T23:26:45.355 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-sklearn. 2026-02-22T23:26:45.363 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../073-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-22T23:26:45.364 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-22T23:26:45.442 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-22T23:26:45.451 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../074-ceph-mgr-diskprediction-local_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T23:26:45.451 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking ceph-mgr-diskprediction-local (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:45.613 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-cachetools. 2026-02-22T23:26:45.621 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../075-python3-cachetools_5.0.0-1_all.deb ... 2026-02-22T23:26:45.621 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-22T23:26:45.637 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-rsa. 2026-02-22T23:26:45.645 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../076-python3-rsa_4.8-1_all.deb ... 2026-02-22T23:26:45.646 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-22T23:26:45.662 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-google-auth. 2026-02-22T23:26:45.671 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../077-python3-google-auth_1.5.1-3_all.deb ... 2026-02-22T23:26:45.671 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-22T23:26:45.689 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-22T23:26:45.697 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-22T23:26:45.697 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-22T23:26:45.713 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-websocket. 2026-02-22T23:26:45.721 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../079-python3-websocket_1.2.3-1_all.deb ... 2026-02-22T23:26:45.721 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-22T23:26:45.739 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-22T23:26:45.747 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../080-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-22T23:26:45.747 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-22T23:26:45.843 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-22T23:26:45.851 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T23:26:45.852 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking ceph-mgr-k8sevents (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:45.867 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-22T23:26:45.875 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../082-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-22T23:26:45.882 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-22T23:26:45.898 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-22T23:26:45.906 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../083-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-22T23:26:45.906 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-22T23:26:45.920 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package jq. 2026-02-22T23:26:45.928 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../084-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-22T23:26:45.929 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-22T23:26:45.944 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package socat. 2026-02-22T23:26:45.952 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../085-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-22T23:26:45.953 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-22T23:26:45.972 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package xmlstarlet. 2026-02-22T23:26:45.980 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-22T23:26:45.980 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-22T23:26:46.015 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package ceph-test. 2026-02-22T23:26:46.023 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../087-ceph-test_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:46.024 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking ceph-test (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:46.553 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package ceph-volume. 2026-02-22T23:26:46.562 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../088-ceph-volume_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T23:26:46.562 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking ceph-volume (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:46.584 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-22T23:26:46.592 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../089-libcephfs-dev_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:46.592 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking libcephfs-dev (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:46.607 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package lua-socket:amd64. 2026-02-22T23:26:46.616 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../090-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2026-02-22T23:26:46.616 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-22T23:26:46.636 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package lua-sec:amd64. 2026-02-22T23:26:46.644 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../091-lua-sec_1.0.2-1_amd64.deb ... 2026-02-22T23:26:46.645 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2026-02-22T23:26:46.661 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package nvme-cli. 2026-02-22T23:26:46.670 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../092-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-22T23:26:46.670 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-22T23:26:46.702 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package pkg-config. 2026-02-22T23:26:46.710 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../093-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2026-02-22T23:26:46.711 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2026-02-22T23:26:46.726 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-22T23:26:46.735 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../094-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-22T23:26:46.735 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-22T23:26:46.763 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-22T23:26:46.771 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../095-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-22T23:26:46.771 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-22T23:26:46.785 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-pastescript. 2026-02-22T23:26:46.793 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../096-python3-pastescript_2.0.2-4_all.deb ... 2026-02-22T23:26:46.794 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2026-02-22T23:26:46.811 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-pluggy. 2026-02-22T23:26:46.818 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../097-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-22T23:26:46.819 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-22T23:26:46.834 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-psutil. 2026-02-22T23:26:46.842 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../098-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-22T23:26:46.843 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-22T23:26:46.861 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-py. 2026-02-22T23:26:46.869 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../099-python3-py_1.10.0-1_all.deb ... 2026-02-22T23:26:46.869 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-22T23:26:46.888 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-pygments. 2026-02-22T23:26:46.896 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../100-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-22T23:26:46.897 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-22T23:26:46.937 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-pyinotify. 2026-02-22T23:26:46.946 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../101-python3-pyinotify_0.9.6-1.3_all.deb ... 2026-02-22T23:26:46.946 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2026-02-22T23:26:46.961 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-toml. 2026-02-22T23:26:46.970 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../102-python3-toml_0.10.2-1_all.deb ... 2026-02-22T23:26:46.970 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-22T23:26:46.984 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-pytest. 2026-02-22T23:26:46.993 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../103-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-22T23:26:46.993 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-22T23:26:47.015 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package python3-simplejson. 2026-02-22T23:26:47.024 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../104-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-22T23:26:47.024 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-22T23:26:47.042 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-22T23:26:47.050 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../105-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-22T23:26:47.050 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-22T23:26:47.120 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package radosgw. 2026-02-22T23:26:47.128 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../106-radosgw_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:47.128 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking radosgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:47.240 INFO:teuthology.orchestra.run.trial115.stdout:Selecting previously unselected package rbd-fuse. 2026-02-22T23:26:47.249 INFO:teuthology.orchestra.run.trial115.stdout:Preparing to unpack .../107-rbd-fuse_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T23:26:47.249 INFO:teuthology.orchestra.run.trial115.stdout:Unpacking rbd-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:47.278 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-22T23:26:47.345 INFO:teuthology.orchestra.run.trial115.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-22T23:26:47.346 INFO:teuthology.orchestra.run.trial115.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-22T23:26:47.404 INFO:teuthology.orchestra.run.trial115.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-22T23:26:47.663 INFO:teuthology.orchestra.run.trial115.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-22T23:26:48.016 INFO:teuthology.orchestra.run.trial115.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-22T23:26:48.026 INFO:teuthology.orchestra.run.trial115.stdout:Setting up cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:48.070 INFO:teuthology.orchestra.run.trial115.stdout:Adding system user cephadm....done 2026-02-22T23:26:48.081 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-22T23:26:48.151 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-22T23:26:48.225 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-22T23:26:48.226 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-22T23:26:48.295 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-22T23:26:48.368 INFO:teuthology.orchestra.run.trial115.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-22T23:26:48.369 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-22T23:26:48.464 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-22T23:26:48.587 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-22T23:26:48.661 INFO:teuthology.orchestra.run.trial115.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2026-02-22T23:26:48.671 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2026-02-22T23:26:48.747 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-22T23:26:48.823 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-ceph-argparse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:48.891 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-22T23:26:48.892 INFO:teuthology.orchestra.run.trial115.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-22T23:26:48.893 INFO:teuthology.orchestra.run.trial115.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-22T23:26:48.894 INFO:teuthology.orchestra.run.trial115.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2026-02-22T23:26:48.895 INFO:teuthology.orchestra.run.trial115.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-22T23:26:48.896 INFO:teuthology.orchestra.run.trial115.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2026-02-22T23:26:48.912 INFO:teuthology.orchestra.run.trial115.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2026-02-22T23:26:48.914 INFO:teuthology.orchestra.run.trial115.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2026-02-22T23:26:48.915 INFO:teuthology.orchestra.run.trial115.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-22T23:26:48.916 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-22T23:26:49.036 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-22T23:26:49.115 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-22T23:26:49.192 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-22T23:26:49.283 INFO:teuthology.orchestra.run.trial115.stdout:Setting up zip (3.0-12build2) ... 2026-02-22T23:26:49.284 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-22T23:26:49.535 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2026-02-22T23:26:49.610 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2026-02-22T23:26:49.611 INFO:teuthology.orchestra.run.trial115.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-22T23:26:49.612 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-22T23:26:49.711 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-22T23:26:49.834 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2026-02-22T23:26:49.960 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-22T23:26:50.041 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-22T23:26:50.144 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-22T23:26:50.218 INFO:teuthology.orchestra.run.trial115.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-22T23:26:50.219 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:50.304 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-22T23:26:50.799 INFO:teuthology.orchestra.run.trial115.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2026-02-22T23:26:50.820 INFO:teuthology.orchestra.run.trial115.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T23:26:50.825 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-22T23:26:50.901 INFO:teuthology.orchestra.run.trial115.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-22T23:26:50.902 INFO:teuthology.orchestra.run.trial115.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-22T23:26:50.903 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-22T23:26:50.976 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-22T23:26:51.047 INFO:teuthology.orchestra.run.trial115.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T23:26:51.048 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-22T23:26:51.127 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2026-02-22T23:26:51.201 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-logutils (0.3.3-8) ... 2026-02-22T23:26:51.278 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-22T23:26:51.351 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2026-02-22T23:26:51.419 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-22T23:26:51.499 INFO:teuthology.orchestra.run.trial115.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-22T23:26:51.500 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-22T23:26:51.581 INFO:teuthology.orchestra.run.trial115.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-22T23:26:51.582 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-22T23:26:51.660 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-22T23:26:51.738 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-22T23:26:51.824 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-22T23:26:51.898 INFO:teuthology.orchestra.run.trial115.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-22T23:26:51.899 INFO:teuthology.orchestra.run.trial115.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2026-02-22T23:26:51.900 INFO:teuthology.orchestra.run.trial115.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-22T23:26:51.901 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-22T23:26:52.034 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2026-02-22T23:26:52.108 INFO:teuthology.orchestra.run.trial115.stdout:Setting up lua-any (27ubuntu1) ... 2026-02-22T23:26:52.109 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-22T23:26:52.184 INFO:teuthology.orchestra.run.trial115.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T23:26:52.185 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-22T23:26:52.270 INFO:teuthology.orchestra.run.trial115.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-22T23:26:52.271 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-webtest (2.0.35-1) ... 2026-02-22T23:26:52.351 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-22T23:26:52.481 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-pastescript (2.0.2-4) ... 2026-02-22T23:26:52.569 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2026-02-22T23:26:52.680 INFO:teuthology.orchestra.run.trial115.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-22T23:26:52.681 INFO:teuthology.orchestra.run.trial115.stdout:Setting up librados2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:52.682 INFO:teuthology.orchestra.run.trial115.stdout:Setting up libsqlite3-mod-ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:52.683 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-22T23:26:53.202 INFO:teuthology.orchestra.run.trial115.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2026-02-22T23:26:53.212 INFO:teuthology.orchestra.run.trial115.stdout:Setting up libcephfs2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:53.213 INFO:teuthology.orchestra.run.trial115.stdout:Setting up libradosstriper1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:53.214 INFO:teuthology.orchestra.run.trial115.stdout:Setting up librbd1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:53.215 INFO:teuthology.orchestra.run.trial115.stdout:Setting up ceph-mgr-modules-core (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:53.216 INFO:teuthology.orchestra.run.trial115.stdout:Setting up ceph-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:53.289 INFO:teuthology.orchestra.run.trial115.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-22T23:26:53.289 INFO:teuthology.orchestra.run.trial115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-22T23:26:53.702 INFO:teuthology.orchestra.run.trial115.stdout:Setting up libcephfs-dev (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:53.703 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-rados (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:53.704 INFO:teuthology.orchestra.run.trial115.stdout:Setting up librgw2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:53.705 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-rbd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:53.705 INFO:teuthology.orchestra.run.trial115.stdout:Setting up rbd-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:53.706 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-rgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:53.707 INFO:teuthology.orchestra.run.trial115.stdout:Setting up python3-cephfs (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:53.708 INFO:teuthology.orchestra.run.trial115.stdout:Setting up ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:53.735 INFO:teuthology.orchestra.run.trial115.stdout:Adding group ceph....done 2026-02-22T23:26:53.772 INFO:teuthology.orchestra.run.trial115.stdout:Adding system user ceph....done 2026-02-22T23:26:53.782 INFO:teuthology.orchestra.run.trial115.stdout:Setting system user ceph properties....done 2026-02-22T23:26:53.788 INFO:teuthology.orchestra.run.trial115.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-22T23:26:53.878 INFO:teuthology.orchestra.run.trial115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-22T23:26:54.152 INFO:teuthology.orchestra.run.trial115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-22T23:26:54.539 INFO:teuthology.orchestra.run.trial115.stdout:Setting up ceph-test (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:54.540 INFO:teuthology.orchestra.run.trial115.stdout:Setting up radosgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:54.810 INFO:teuthology.orchestra.run.trial115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-22T23:26:54.810 INFO:teuthology.orchestra.run.trial115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-22T23:26:55.202 INFO:teuthology.orchestra.run.trial115.stdout:Setting up ceph-base (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:55.322 INFO:teuthology.orchestra.run.trial115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-22T23:26:55.712 INFO:teuthology.orchestra.run.trial115.stdout:Setting up ceph-mds (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:55.772 INFO:teuthology.orchestra.run.trial115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-22T23:26:55.772 INFO:teuthology.orchestra.run.trial115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-22T23:26:56.166 INFO:teuthology.orchestra.run.trial115.stdout:Setting up ceph-mgr (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:56.223 INFO:teuthology.orchestra.run.trial115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-22T23:26:56.223 INFO:teuthology.orchestra.run.trial115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-22T23:26:56.595 INFO:teuthology.orchestra.run.trial115.stdout:Setting up ceph-osd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:56.680 INFO:teuthology.orchestra.run.trial115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-22T23:26:56.680 INFO:teuthology.orchestra.run.trial115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-22T23:26:57.066 INFO:teuthology.orchestra.run.trial115.stdout:Setting up ceph-mgr-k8sevents (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:57.067 INFO:teuthology.orchestra.run.trial115.stdout:Setting up ceph-mgr-diskprediction-local (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:57.077 INFO:teuthology.orchestra.run.trial115.stdout:Setting up ceph-mon (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:57.149 INFO:teuthology.orchestra.run.trial115.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-22T23:26:57.149 INFO:teuthology.orchestra.run.trial115.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-22T23:26:57.558 INFO:teuthology.orchestra.run.trial115.stdout:Setting up ceph-mgr-cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:57.571 INFO:teuthology.orchestra.run.trial115.stdout:Setting up ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:57.572 INFO:teuthology.orchestra.run.trial115.stdout:Setting up ceph-mgr-dashboard (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:57.582 INFO:teuthology.orchestra.run.trial115.stdout:Setting up ceph-volume (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T23:26:57.694 INFO:teuthology.orchestra.run.trial115.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2026-02-22T23:26:57.703 INFO:teuthology.orchestra.run.trial115.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-22T23:26:57.946 INFO:teuthology.orchestra.run.trial115.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-22T23:26:58.029 INFO:teuthology.orchestra.run.trial115.stdout:Processing triggers for install-info (6.8-4build1) ... 2026-02-22T23:26:58.484 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:26:58.485 INFO:teuthology.orchestra.run.trial115.stdout:Running kernel seems to be up-to-date. 2026-02-22T23:26:58.485 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:26:58.485 INFO:teuthology.orchestra.run.trial115.stdout:The processor microcode seems to be up-to-date. 2026-02-22T23:26:58.485 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:26:58.485 INFO:teuthology.orchestra.run.trial115.stdout:No services need to be restarted. 2026-02-22T23:26:58.485 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:26:58.485 INFO:teuthology.orchestra.run.trial115.stdout:No containers need to be restarted. 2026-02-22T23:26:58.485 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:26:58.485 INFO:teuthology.orchestra.run.trial115.stdout:No user sessions are running outdated binaries. 2026-02-22T23:26:58.486 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:26:58.486 INFO:teuthology.orchestra.run.trial115.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-22T23:26:59.281 INFO:teuthology.orchestra.run.trial115.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-22T23:26:59.287 DEBUG:teuthology.parallel:result is None 2026-02-22T23:26:59.288 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T23:26:59.442 DEBUG:teuthology.orchestra.run.trial115:> dpkg-query -W -f '${Version}' ceph 2026-02-22T23:26:59.453 INFO:teuthology.orchestra.run.trial115.stdout:19.2.3-689-gf90f4eae-1jammy 2026-02-22T23:26:59.453 INFO:teuthology.packaging:The installed version of ceph is 19.2.3-689-gf90f4eae-1jammy 2026-02-22T23:26:59.453 INFO:teuthology.task.install:The correct ceph version 19.2.3-689-gf90f4eae-1jammy is installed. 2026-02-22T23:26:59.455 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-22T23:26:59.455 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:26:59.455 DEBUG:teuthology.orchestra.run.trial115:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-22T23:26:59.503 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-22T23:26:59.504 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:26:59.504 DEBUG:teuthology.orchestra.run.trial115:> sudo dd of=/usr/bin/daemon-helper 2026-02-22T23:26:59.555 DEBUG:teuthology.orchestra.run.trial115:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-22T23:26:59.603 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-22T23:26:59.603 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:26:59.603 DEBUG:teuthology.orchestra.run.trial115:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-22T23:26:59.651 DEBUG:teuthology.orchestra.run.trial115:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-22T23:26:59.699 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-22T23:26:59.699 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:26:59.699 DEBUG:teuthology.orchestra.run.trial115:> sudo dd of=/usr/bin/stdin-killer 2026-02-22T23:26:59.748 DEBUG:teuthology.orchestra.run.trial115:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-22T23:26:59.799 INFO:teuthology.run_tasks:Running task ceph... 2026-02-22T23:26:59.911 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-22T23:26:59.911 DEBUG:teuthology.orchestra.run.trial115:> sudo chmod 777 /var/log/ceph 2026-02-22T23:26:59.919 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-22T23:26:59.919 DEBUG:teuthology.orchestra.run.trial115:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-22T23:26:59.970 INFO:tasks.ceph:Creating extra log directories... 2026-02-22T23:26:59.971 DEBUG:teuthology.orchestra.run.trial115:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-22T23:27:00.028 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-22T23:27:00.028 INFO:tasks.ceph:config {'conf': {'osd': {'osd max pg log entries': 5, 'osd min pg log entries': 5, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(OBJECT_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(SLOW_OPS\\)', 'deep-scrub', 'missing', 'overall HEALTH_', 'repair', 'slow request', 'unfound', '\\(POOL_APP_NOT_ENABLED\\)', 'enough copies available', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-22T23:27:00.028 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64402', 'branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'description': 'rados/singleton/{all/ec-inconsistent-hinfo mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'flavor': 'default', 'job_id': '64402', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'trial', 'name': 'yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri6-testing-2026-02-22-1629-squid'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}}, 'owner': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 5907, 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri6-testing-2026-02-22-1629-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'targets': {'trial115.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKcSulXW+clEv4dzDKLT66gn41AhCyV7P0t88DikHKhFHCNG6AZtHU3YqyTLqj9M7wqZX8mvVaWcU3WYT27puKQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'osd max pg log entries': 5, 'osd min pg log entries': 5, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'create_rbd_pool': False, 'log-ignorelist': ['\\(OBJECT_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(SLOW_OPS\\)', 'deep-scrub', 'missing', 'overall HEALTH_', 'repair', 'slow request', 'unfound', '\\(POOL_APP_NOT_ENABLED\\)', 'enough copies available', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force'], 'flavor': 'default', 'fs': 'xfs', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'cluster': 'ceph'}}, {'ec_inconsistent_hinfo': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-22_19:35:40', 'tube': 'trial', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653'} 2026-02-22T23:27:00.028 DEBUG:teuthology.orchestra.run.trial115:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-22T23:27:00.063 DEBUG:teuthology.orchestra.run.trial115:> sudo install -d -m0777 -- /var/run/ceph 2026-02-22T23:27:00.111 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:27:00.111 DEBUG:teuthology.orchestra.run.trial115:> dd if=/scratch_devs of=/dev/stdout 2026-02-22T23:27:00.159 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-22T23:27:00.160 DEBUG:teuthology.orchestra.run.trial115:> stat /dev/vg_nvme/lv_1 2026-02-22T23:27:00.207 INFO:teuthology.orchestra.run.trial115.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-22T23:27:00.207 INFO:teuthology.orchestra.run.trial115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T23:27:00.208 INFO:teuthology.orchestra.run.trial115.stdout:Device: 5h/5d Inode: 906 Links: 1 2026-02-22T23:27:00.208 INFO:teuthology.orchestra.run.trial115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T23:27:00.208 INFO:teuthology.orchestra.run.trial115.stdout:Access: 2026-02-22 23:26:16.989957068 +0000 2026-02-22T23:27:00.208 INFO:teuthology.orchestra.run.trial115.stdout:Modify: 2026-02-22 23:26:16.821959923 +0000 2026-02-22T23:27:00.208 INFO:teuthology.orchestra.run.trial115.stdout:Change: 2026-02-22 23:26:16.821959923 +0000 2026-02-22T23:27:00.208 INFO:teuthology.orchestra.run.trial115.stdout: Birth: - 2026-02-22T23:27:00.208 DEBUG:teuthology.orchestra.run.trial115:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-22T23:27:00.258 INFO:teuthology.orchestra.run.trial115.stderr:1+0 records in 2026-02-22T23:27:00.258 INFO:teuthology.orchestra.run.trial115.stderr:1+0 records out 2026-02-22T23:27:00.258 INFO:teuthology.orchestra.run.trial115.stderr:512 bytes copied, 0.000192789 s, 2.7 MB/s 2026-02-22T23:27:00.260 DEBUG:teuthology.orchestra.run.trial115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-22T23:27:00.308 DEBUG:teuthology.orchestra.run.trial115:> stat /dev/vg_nvme/lv_2 2026-02-22T23:27:00.354 INFO:teuthology.orchestra.run.trial115.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-22T23:27:00.354 INFO:teuthology.orchestra.run.trial115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T23:27:00.354 INFO:teuthology.orchestra.run.trial115.stdout:Device: 5h/5d Inode: 929 Links: 1 2026-02-22T23:27:00.354 INFO:teuthology.orchestra.run.trial115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T23:27:00.354 INFO:teuthology.orchestra.run.trial115.stdout:Access: 2026-02-22 23:26:17.325951353 +0000 2026-02-22T23:27:00.354 INFO:teuthology.orchestra.run.trial115.stdout:Modify: 2026-02-22 23:26:17.137954550 +0000 2026-02-22T23:27:00.354 INFO:teuthology.orchestra.run.trial115.stdout:Change: 2026-02-22 23:26:17.137954550 +0000 2026-02-22T23:27:00.354 INFO:teuthology.orchestra.run.trial115.stdout: Birth: - 2026-02-22T23:27:00.355 DEBUG:teuthology.orchestra.run.trial115:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-22T23:27:00.402 INFO:teuthology.orchestra.run.trial115.stderr:1+0 records in 2026-02-22T23:27:00.402 INFO:teuthology.orchestra.run.trial115.stderr:1+0 records out 2026-02-22T23:27:00.402 INFO:teuthology.orchestra.run.trial115.stderr:512 bytes copied, 0.000181738 s, 2.8 MB/s 2026-02-22T23:27:00.403 DEBUG:teuthology.orchestra.run.trial115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-22T23:27:00.453 DEBUG:teuthology.orchestra.run.trial115:> stat /dev/vg_nvme/lv_3 2026-02-22T23:27:00.499 INFO:teuthology.orchestra.run.trial115.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-22T23:27:00.499 INFO:teuthology.orchestra.run.trial115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T23:27:00.500 INFO:teuthology.orchestra.run.trial115.stdout:Device: 5h/5d Inode: 952 Links: 1 2026-02-22T23:27:00.500 INFO:teuthology.orchestra.run.trial115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T23:27:00.500 INFO:teuthology.orchestra.run.trial115.stdout:Access: 2026-02-22 23:26:17.685945234 +0000 2026-02-22T23:27:00.500 INFO:teuthology.orchestra.run.trial115.stdout:Modify: 2026-02-22 23:26:17.501948361 +0000 2026-02-22T23:27:00.500 INFO:teuthology.orchestra.run.trial115.stdout:Change: 2026-02-22 23:26:17.501948361 +0000 2026-02-22T23:27:00.500 INFO:teuthology.orchestra.run.trial115.stdout: Birth: - 2026-02-22T23:27:00.500 DEBUG:teuthology.orchestra.run.trial115:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-22T23:27:00.550 INFO:teuthology.orchestra.run.trial115.stderr:1+0 records in 2026-02-22T23:27:00.550 INFO:teuthology.orchestra.run.trial115.stderr:1+0 records out 2026-02-22T23:27:00.550 INFO:teuthology.orchestra.run.trial115.stderr:512 bytes copied, 0.000194632 s, 2.6 MB/s 2026-02-22T23:27:00.552 DEBUG:teuthology.orchestra.run.trial115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-22T23:27:00.600 DEBUG:teuthology.orchestra.run.trial115:> stat /dev/vg_nvme/lv_4 2026-02-22T23:27:00.646 INFO:teuthology.orchestra.run.trial115.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-22T23:27:00.646 INFO:teuthology.orchestra.run.trial115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-22T23:27:00.646 INFO:teuthology.orchestra.run.trial115.stdout:Device: 5h/5d Inode: 976 Links: 1 2026-02-22T23:27:00.646 INFO:teuthology.orchestra.run.trial115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-22T23:27:00.646 INFO:teuthology.orchestra.run.trial115.stdout:Access: 2026-02-22 23:26:17.865942173 +0000 2026-02-22T23:27:00.646 INFO:teuthology.orchestra.run.trial115.stdout:Modify: 2026-02-22 23:26:17.861942242 +0000 2026-02-22T23:27:00.646 INFO:teuthology.orchestra.run.trial115.stdout:Change: 2026-02-22 23:26:17.861942242 +0000 2026-02-22T23:27:00.647 INFO:teuthology.orchestra.run.trial115.stdout: Birth: - 2026-02-22T23:27:00.647 DEBUG:teuthology.orchestra.run.trial115:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-22T23:27:00.693 INFO:teuthology.orchestra.run.trial115.stderr:1+0 records in 2026-02-22T23:27:00.694 INFO:teuthology.orchestra.run.trial115.stderr:1+0 records out 2026-02-22T23:27:00.694 INFO:teuthology.orchestra.run.trial115.stderr:512 bytes copied, 0.000171308 s, 3.0 MB/s 2026-02-22T23:27:00.694 DEBUG:teuthology.orchestra.run.trial115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-22T23:27:00.741 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-22T23:27:00.741 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial115.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2026-02-22T23:27:00.741 INFO:tasks.ceph:Generating config... 2026-02-22T23:27:00.743 INFO:tasks.ceph:[osd] osd max pg log entries = 5 2026-02-22T23:27:00.743 INFO:tasks.ceph:[osd] osd min pg log entries = 5 2026-02-22T23:27:00.744 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-02-22T23:27:00.744 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2026-02-22T23:27:00.744 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2026-02-22T23:27:00.744 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-02-22T23:27:00.744 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2026-02-22T23:27:00.744 INFO:tasks.ceph:[osd] debug bluefs = 20 2026-02-22T23:27:00.744 INFO:tasks.ceph:[osd] debug bluestore = 20 2026-02-22T23:27:00.744 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-22T23:27:00.744 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-22T23:27:00.744 INFO:tasks.ceph:[osd] debug rocksdb = 10 2026-02-22T23:27:00.744 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-02-22T23:27:00.744 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-02-22T23:27:00.745 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-02-22T23:27:00.745 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2026-02-22T23:27:00.745 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2026-02-22T23:27:00.745 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-02-22T23:27:00.745 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2026-02-22T23:27:00.745 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2026-02-22T23:27:00.745 INFO:tasks.ceph:[osd] osd mclock skip benchmark = True 2026-02-22T23:27:00.745 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-02-22T23:27:00.745 INFO:tasks.ceph:[osd] osd op queue = debug_random 2026-02-22T23:27:00.745 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2026-02-22T23:27:00.745 INFO:tasks.ceph:[global] mon election default strategy = 1 2026-02-22T23:27:00.745 INFO:tasks.ceph:[global] ms bind msgr1 = False 2026-02-22T23:27:00.746 INFO:tasks.ceph:[global] ms bind msgr2 = True 2026-02-22T23:27:00.746 INFO:tasks.ceph:[global] ms type = async 2026-02-22T23:27:00.746 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-22T23:27:00.746 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-22T23:27:00.746 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-22T23:27:00.746 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-22T23:27:00.746 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-22T23:27:00.746 INFO:tasks.ceph:[mon] mon scrub interval = 300 2026-02-22T23:27:00.746 INFO:tasks.ceph:Setting up mon.a... 2026-02-22T23:27:00.746 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-22T23:27:00.800 INFO:teuthology.orchestra.run.trial115.stdout:creating /etc/ceph/ceph.keyring 2026-02-22T23:27:00.803 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-22T23:27:00.862 DEBUG:teuthology.orchestra.run.trial115:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-22T23:27:00.911 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.115'), ('mon.b', '[v2:10.20.193.115:3301,v1:10.20.193.115:6790]'), ('mon.c', '[v2:10.20.193.115:3302,v1:10.20.193.115:6791]')] 2026-02-22T23:27:00.912 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.115,[v2:10.20.193.115:3301,v1:10.20.193.115:6790],[v2:10.20.193.115:3302,v1:10.20.193.115:6791]', 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'osd max pg log entries': 5, 'osd min pg log entries': 5, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2026-02-22T23:27:00.913 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:27:00.913 DEBUG:teuthology.orchestra.run.trial115:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-22T23:27:00.956 DEBUG:teuthology.orchestra.run.trial115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.115 --addv b '[v2:10.20.193.115:3301,v1:10.20.193.115:6790]' --addv c '[v2:10.20.193.115:3302,v1:10.20.193.115:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2026-02-22T23:27:01.018 INFO:teuthology.orchestra.run.trial115.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-22T23:27:01.018 INFO:teuthology.orchestra.run.trial115.stdout:monmaptool: generated fsid a4a3ada6-bd8c-4e08-a984-e18b05a50ce6 2026-02-22T23:27:01.018 INFO:teuthology.orchestra.run.trial115.stdout:setting min_mon_release = quincy 2026-02-22T23:27:01.018 INFO:teuthology.orchestra.run.trial115.stdout:epoch 0 2026-02-22T23:27:01.018 INFO:teuthology.orchestra.run.trial115.stdout:fsid a4a3ada6-bd8c-4e08-a984-e18b05a50ce6 2026-02-22T23:27:01.018 INFO:teuthology.orchestra.run.trial115.stdout:last_changed 2026-02-22T23:27:01.020831+0000 2026-02-22T23:27:01.018 INFO:teuthology.orchestra.run.trial115.stdout:created 2026-02-22T23:27:01.020831+0000 2026-02-22T23:27:01.018 INFO:teuthology.orchestra.run.trial115.stdout:min_mon_release 17 (quincy) 2026-02-22T23:27:01.018 INFO:teuthology.orchestra.run.trial115.stdout:election_strategy: 1 2026-02-22T23:27:01.018 INFO:teuthology.orchestra.run.trial115.stdout:0: [v2:10.20.193.115:3300/0,v1:10.20.193.115:6789/0] mon.a 2026-02-22T23:27:01.018 INFO:teuthology.orchestra.run.trial115.stdout:1: [v2:10.20.193.115:3301/0,v1:10.20.193.115:6790/0] mon.b 2026-02-22T23:27:01.019 INFO:teuthology.orchestra.run.trial115.stdout:2: [v2:10.20.193.115:3302/0,v1:10.20.193.115:6791/0] mon.c 2026-02-22T23:27:01.019 INFO:teuthology.orchestra.run.trial115.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2026-02-22T23:27:01.019 DEBUG:teuthology.orchestra.run.trial115:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-22T23:27:01.066 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a4a3ada6-bd8c-4e08-a984-e18b05a50ce6... 2026-02-22T23:27:01.066 DEBUG:teuthology.orchestra.run.trial115:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-22T23:27:01.127 INFO:teuthology.orchestra.run.trial115.stdout:[global] 2026-02-22T23:27:01.127 INFO:teuthology.orchestra.run.trial115.stdout: chdir = "" 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: auth supported = cephx 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: filestore xattr use omap = true 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: mon clock drift allowed = 1.000 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: osd crush chooseleaf type = 0 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: auth debug = true 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: ms die on old message = true 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: ms die on bug = true 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: mon pg warn max object skew = 0 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: # disable pg_autoscaler by default for new pools 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: osd pool default size = 2 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.128 INFO:teuthology.orchestra.run.trial115.stdout: mon osd allow primary affinity = true 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: mon osd allow pg remap = true 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: mon warn on legacy crush tunables = false 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: mon warn on crush straw calc version zero = false 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: mon warn on no sortbitwise = false 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: mon warn on osd down out interval zero = false 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: mon warn on too few osds = false 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: mon_allow_pool_size_one = true 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: osd default data pool replay window = 5 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: mon allow pool delete = true 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: mon cluster log file level = debug 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: debug asserts on shutdown = true 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: mon health detail to clog = false 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: mon host = "10.20.193.115,[v2:10.20.193.115:3301,v1:10.20.193.115:6790],[v2:10.20.193.115:3302,v1:10.20.193.115:6791]" 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: mon election default strategy = 1 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: ms bind msgr1 = False 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: ms bind msgr2 = True 2026-02-22T23:27:01.129 INFO:teuthology.orchestra.run.trial115.stdout: ms type = async 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: fsid = a4a3ada6-bd8c-4e08-a984-e18b05a50ce6 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout:[osd] 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: osd journal size = 100 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: osd scrub load threshold = 5.0 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: osd scrub max interval = 600 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: osd mclock profile = high_recovery_ops 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: osd mclock skip benchmark = True 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: osd recover clone overlap = true 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: osd recovery max chunk = 1048576 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: osd debug shutdown = true 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: osd debug op order = true 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: osd debug verify stray on activate = true 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: osd debug trim objects = true 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: osd open classes on start = true 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: osd debug pg log writeout = true 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: osd deep scrub update digest min age = 30 2026-02-22T23:27:01.130 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: osd map max advance = 10 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: journal zero on create = true 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: filestore ondisk finisher threads = 3 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: filestore apply finisher threads = 3 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: bdev debug aio = true 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: osd debug misdirected ops = true 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: osd max pg log entries = 5 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: osd min pg log entries = 5 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: bluestore block size = 96636764160 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: bluestore compression algorithm = zstd 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: bluestore compression mode = aggressive 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: bluestore fsck on mount = True 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: bluestore zero block detection = True 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: debug bluefs = 20 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: debug bluestore = 20 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: debug ms = 1 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: debug osd = 20 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: debug rocksdb = 10 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: mon osd backfillfull_ratio = 0.85 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: mon osd full ratio = 0.9 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: mon osd nearfull ratio = 0.8 2026-02-22T23:27:01.131 INFO:teuthology.orchestra.run.trial115.stdout: osd debug verify cached snaps = True 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: osd debug verify missing on start = True 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: osd failsafe full ratio = 0.95 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: osd mclock override recovery settings = True 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: osd objectstore = bluestore 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: osd op queue = debug_random 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: osd op queue cut off = debug_random 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout:[mgr] 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: debug ms = 1 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: debug mgr = 20 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: debug mon = 20 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: debug auth = 20 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: mon reweight min pgs per osd = 4 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: mon reweight min bytes per osd = 10 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: mgr/telemetry/nag = false 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout:[mon] 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: debug ms = 1 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: debug mon = 20 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: debug paxos = 20 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: debug auth = 20 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: mon data avail warn = 5 2026-02-22T23:27:01.132 INFO:teuthology.orchestra.run.trial115.stdout: mon mgr mkfs grace = 240 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: mon reweight min pgs per osd = 4 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: mon osd reporter subtree level = osd 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: mon osd prime pg temp = true 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: mon reweight min bytes per osd = 10 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: auth mon ticket ttl = 660 # 11m 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: auth service ticket ttl = 240 # 4m 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: # don't complain about insecure global_id in the test suite 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: # 1m isn't quite enough 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: mon_down_mkfs_grace = 2m 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: mon_warn_on_filestore_osds = false 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: mon scrub interval = 300 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout:[client] 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: rgw cache enabled = true 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: rgw enable ops log = true 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: rgw enable usage log = true 2026-02-22T23:27:01.133 INFO:teuthology.orchestra.run.trial115.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-22T23:27:01.134 INFO:teuthology.orchestra.run.trial115.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-22T23:27:01.134 INFO:teuthology.orchestra.run.trial115.stdout:[mon.a] 2026-02-22T23:27:01.134 INFO:teuthology.orchestra.run.trial115.stdout:[mon.b] 2026-02-22T23:27:01.134 INFO:teuthology.orchestra.run.trial115.stdout:[mon.c] 2026-02-22T23:27:01.136 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-22T23:27:01.136 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-22T23:27:01.201 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-22T23:27:01.202 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:27:01.202 DEBUG:teuthology.orchestra.run.trial115:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-22T23:27:01.246 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:27:01.247 DEBUG:teuthology.orchestra.run.trial115:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-22T23:27:01.291 INFO:tasks.ceph:Sending monmap to node ubuntu@trial115.front.sepia.ceph.com 2026-02-22T23:27:01.291 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:27:01.291 DEBUG:teuthology.orchestra.run.trial115:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-22T23:27:01.291 DEBUG:teuthology.orchestra.run.trial115:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-22T23:27:01.345 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:27:01.345 DEBUG:teuthology.orchestra.run.trial115:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-22T23:27:01.391 INFO:tasks.ceph:Setting up mon nodes... 2026-02-22T23:27:01.391 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-22T23:27:01.391 DEBUG:teuthology.orchestra.run.trial115:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-22T23:27:01.458 INFO:teuthology.orchestra.run.trial115.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-22T23:27:01.460 INFO:tasks.ceph:Setting up mds nodes... 2026-02-22T23:27:01.461 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-22T23:27:01.461 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-22T23:27:01.461 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial115.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2026-02-22T23:27:01.461 DEBUG:teuthology.orchestra.run.trial115:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-22T23:27:01.511 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-22T23:27:01.511 INFO:tasks.ceph:role: osd.0 2026-02-22T23:27:01.511 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial115.front.sepia.ceph.com 2026-02-22T23:27:01.511 DEBUG:teuthology.orchestra.run.trial115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-22T23:27:01.564 INFO:teuthology.orchestra.run.trial115.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T23:27:01.564 INFO:teuthology.orchestra.run.trial115.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T23:27:01.564 INFO:teuthology.orchestra.run.trial115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T23:27:01.564 INFO:teuthology.orchestra.run.trial115.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T23:27:01.564 INFO:teuthology.orchestra.run.trial115.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T23:27:01.564 INFO:teuthology.orchestra.run.trial115.stdout: = sunit=0 swidth=0 blks 2026-02-22T23:27:01.564 INFO:teuthology.orchestra.run.trial115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T23:27:01.564 INFO:teuthology.orchestra.run.trial115.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T23:27:01.564 INFO:teuthology.orchestra.run.trial115.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T23:27:01.564 INFO:teuthology.orchestra.run.trial115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T23:27:02.139 INFO:teuthology.orchestra.run.trial115.stdout:Discarding blocks...Done. 2026-02-22T23:27:02.141 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial115.front.sepia.ceph.com -o noatime 2026-02-22T23:27:02.141 DEBUG:teuthology.orchestra.run.trial115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-22T23:27:02.196 DEBUG:teuthology.orchestra.run.trial115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-22T23:27:02.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T23:27:02.246 INFO:teuthology.orchestra.run.trial115.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T23:27:02.246 DEBUG:teuthology.orchestra.run.trial115:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-22T23:27:02.296 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-22T23:27:02.296 INFO:tasks.ceph:role: osd.1 2026-02-22T23:27:02.296 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial115.front.sepia.ceph.com 2026-02-22T23:27:02.296 DEBUG:teuthology.orchestra.run.trial115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-22T23:27:02.349 INFO:teuthology.orchestra.run.trial115.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T23:27:02.349 INFO:teuthology.orchestra.run.trial115.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T23:27:02.349 INFO:teuthology.orchestra.run.trial115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T23:27:02.349 INFO:teuthology.orchestra.run.trial115.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T23:27:02.350 INFO:teuthology.orchestra.run.trial115.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T23:27:02.350 INFO:teuthology.orchestra.run.trial115.stdout: = sunit=0 swidth=0 blks 2026-02-22T23:27:02.350 INFO:teuthology.orchestra.run.trial115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T23:27:02.350 INFO:teuthology.orchestra.run.trial115.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T23:27:02.350 INFO:teuthology.orchestra.run.trial115.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T23:27:02.350 INFO:teuthology.orchestra.run.trial115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T23:27:02.914 INFO:teuthology.orchestra.run.trial115.stdout:Discarding blocks...Done. 2026-02-22T23:27:02.916 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial115.front.sepia.ceph.com -o noatime 2026-02-22T23:27:02.916 DEBUG:teuthology.orchestra.run.trial115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-22T23:27:02.973 DEBUG:teuthology.orchestra.run.trial115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-22T23:27:03.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T23:27:03.022 INFO:teuthology.orchestra.run.trial115.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T23:27:03.022 DEBUG:teuthology.orchestra.run.trial115:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-22T23:27:03.071 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-22T23:27:03.072 INFO:tasks.ceph:role: osd.2 2026-02-22T23:27:03.072 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial115.front.sepia.ceph.com 2026-02-22T23:27:03.072 DEBUG:teuthology.orchestra.run.trial115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-22T23:27:03.125 INFO:teuthology.orchestra.run.trial115.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T23:27:03.126 INFO:teuthology.orchestra.run.trial115.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T23:27:03.126 INFO:teuthology.orchestra.run.trial115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T23:27:03.126 INFO:teuthology.orchestra.run.trial115.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T23:27:03.126 INFO:teuthology.orchestra.run.trial115.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T23:27:03.126 INFO:teuthology.orchestra.run.trial115.stdout: = sunit=0 swidth=0 blks 2026-02-22T23:27:03.126 INFO:teuthology.orchestra.run.trial115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T23:27:03.126 INFO:teuthology.orchestra.run.trial115.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T23:27:03.126 INFO:teuthology.orchestra.run.trial115.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T23:27:03.126 INFO:teuthology.orchestra.run.trial115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T23:27:03.687 INFO:teuthology.orchestra.run.trial115.stdout:Discarding blocks...Done. 2026-02-22T23:27:03.689 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial115.front.sepia.ceph.com -o noatime 2026-02-22T23:27:03.689 DEBUG:teuthology.orchestra.run.trial115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-22T23:27:03.744 DEBUG:teuthology.orchestra.run.trial115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-22T23:27:03.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T23:27:03.794 INFO:teuthology.orchestra.run.trial115.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T23:27:03.794 DEBUG:teuthology.orchestra.run.trial115:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2026-02-22T23:27:03.843 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-22T23:27:03.843 INFO:tasks.ceph:role: osd.3 2026-02-22T23:27:03.843 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@trial115.front.sepia.ceph.com 2026-02-22T23:27:03.843 DEBUG:teuthology.orchestra.run.trial115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2026-02-22T23:27:03.893 INFO:teuthology.orchestra.run.trial115.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=45875200 blks 2026-02-22T23:27:03.893 INFO:teuthology.orchestra.run.trial115.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-22T23:27:03.893 INFO:teuthology.orchestra.run.trial115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-22T23:27:03.893 INFO:teuthology.orchestra.run.trial115.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-22T23:27:03.893 INFO:teuthology.orchestra.run.trial115.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-22T23:27:03.893 INFO:teuthology.orchestra.run.trial115.stdout: = sunit=0 swidth=0 blks 2026-02-22T23:27:03.893 INFO:teuthology.orchestra.run.trial115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-22T23:27:03.893 INFO:teuthology.orchestra.run.trial115.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-22T23:27:03.893 INFO:teuthology.orchestra.run.trial115.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-22T23:27:03.893 INFO:teuthology.orchestra.run.trial115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-22T23:27:04.454 INFO:teuthology.orchestra.run.trial115.stdout:Discarding blocks...Done. 2026-02-22T23:27:04.456 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@trial115.front.sepia.ceph.com -o noatime 2026-02-22T23:27:04.457 DEBUG:teuthology.orchestra.run.trial115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2026-02-22T23:27:04.516 DEBUG:teuthology.orchestra.run.trial115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2026-02-22T23:27:04.564 INFO:teuthology.orchestra.run.trial115.stderr:sudo: /sbin/restorecon: command not found 2026-02-22T23:27:04.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T23:27:04.565 DEBUG:teuthology.orchestra.run.trial115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T23:27:04.628 INFO:teuthology.orchestra.run.trial115.stderr:2026-02-22T23:27:04.626+0000 7faacc6438c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2026-02-22T23:27:04.628 INFO:teuthology.orchestra.run.trial115.stderr:2026-02-22T23:27:04.626+0000 7faacc6438c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2026-02-22T23:27:04.628 INFO:teuthology.orchestra.run.trial115.stderr:2026-02-22T23:27:04.626+0000 7faacc6438c0 -1 bdev(0x55b875249800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2026-02-22T23:27:04.628 INFO:teuthology.orchestra.run.trial115.stderr:2026-02-22T23:27:04.626+0000 7faacc6438c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2026-02-22T23:27:07.763 DEBUG:teuthology.orchestra.run.trial115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-22T23:27:07.816 DEBUG:teuthology.orchestra.run.trial115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T23:27:07.880 INFO:teuthology.orchestra.run.trial115.stderr:2026-02-22T23:27:07.878+0000 7fbd42bcc8c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2026-02-22T23:27:07.881 INFO:teuthology.orchestra.run.trial115.stderr:2026-02-22T23:27:07.878+0000 7fbd42bcc8c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2026-02-22T23:27:07.881 INFO:teuthology.orchestra.run.trial115.stderr:2026-02-22T23:27:07.878+0000 7fbd42bcc8c0 -1 bdev(0x557488e53800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2026-02-22T23:27:07.881 INFO:teuthology.orchestra.run.trial115.stderr:2026-02-22T23:27:07.878+0000 7fbd42bcc8c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2026-02-22T23:27:10.982 DEBUG:teuthology.orchestra.run.trial115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-22T23:27:11.032 DEBUG:teuthology.orchestra.run.trial115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T23:27:11.095 INFO:teuthology.orchestra.run.trial115.stderr:2026-02-22T23:27:11.094+0000 7f7b5dab58c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2026-02-22T23:27:11.096 INFO:teuthology.orchestra.run.trial115.stderr:2026-02-22T23:27:11.094+0000 7f7b5dab58c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2026-02-22T23:27:11.096 INFO:teuthology.orchestra.run.trial115.stderr:2026-02-22T23:27:11.094+0000 7f7b5dab58c0 -1 bdev(0x55572a77f800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2026-02-22T23:27:11.096 INFO:teuthology.orchestra.run.trial115.stderr:2026-02-22T23:27:11.094+0000 7f7b5dab58c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2026-02-22T23:27:14.235 DEBUG:teuthology.orchestra.run.trial115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-22T23:27:14.284 DEBUG:teuthology.orchestra.run.trial115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-22T23:27:14.348 INFO:teuthology.orchestra.run.trial115.stderr:2026-02-22T23:27:14.346+0000 7fdf55aec8c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2026-02-22T23:27:14.349 INFO:teuthology.orchestra.run.trial115.stderr:2026-02-22T23:27:14.346+0000 7fdf55aec8c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2026-02-22T23:27:14.349 INFO:teuthology.orchestra.run.trial115.stderr:2026-02-22T23:27:14.346+0000 7fdf55aec8c0 -1 bdev(0x55b9ce68f800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2026-02-22T23:27:14.349 INFO:teuthology.orchestra.run.trial115.stderr:2026-02-22T23:27:14.346+0000 7fdf55aec8c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2026-02-22T23:27:17.436 DEBUG:teuthology.orchestra.run.trial115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-22T23:27:17.488 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-22T23:27:17.488 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:27:17.488 DEBUG:teuthology.orchestra.run.trial115:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-22T23:27:17.540 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:27:17.540 DEBUG:teuthology.orchestra.run.trial115:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-22T23:27:17.591 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:27:17.592 DEBUG:teuthology.orchestra.run.trial115:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-22T23:27:17.644 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:27:17.644 DEBUG:teuthology.orchestra.run.trial115:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-22T23:27:17.696 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:27:17.696 DEBUG:teuthology.orchestra.run.trial115:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2026-02-22T23:27:17.748 INFO:tasks.ceph:Adding keys to all mons... 2026-02-22T23:27:17.748 DEBUG:teuthology.orchestra.run.trial115:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-22T23:27:17.797 INFO:teuthology.orchestra.run.trial115.stdout:[mgr.x] 2026-02-22T23:27:17.797 INFO:teuthology.orchestra.run.trial115.stdout: key = AQDFkJtpmrx2GxAAnryp5z62fnLTvJ/CxN02EQ== 2026-02-22T23:27:17.797 INFO:teuthology.orchestra.run.trial115.stdout:[osd.0] 2026-02-22T23:27:17.797 INFO:teuthology.orchestra.run.trial115.stdout: key = AQDIkJtpB/qRJRAAJWSKFO7+P94h+zo62ZrJRQ== 2026-02-22T23:27:17.797 INFO:teuthology.orchestra.run.trial115.stdout:[osd.1] 2026-02-22T23:27:17.797 INFO:teuthology.orchestra.run.trial115.stdout: key = AQDLkJtp9xmfNBAAFSpcI5R7Kwaxz+sEuZAJgg== 2026-02-22T23:27:17.797 INFO:teuthology.orchestra.run.trial115.stdout:[osd.2] 2026-02-22T23:27:17.797 INFO:teuthology.orchestra.run.trial115.stdout: key = AQDPkJtp9dHZBRAAs42cUnczGPj88qQGtYcGZA== 2026-02-22T23:27:17.797 INFO:teuthology.orchestra.run.trial115.stdout:[osd.3] 2026-02-22T23:27:17.797 INFO:teuthology.orchestra.run.trial115.stdout: key = AQDSkJtpN3zqFBAA9bCuTHdtyo6ZhJeYZFK8KQ== 2026-02-22T23:27:17.798 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-22T23:27:17.860 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T23:27:17.923 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T23:27:17.988 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T23:27:18.051 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-22T23:27:18.115 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-22T23:27:18.115 DEBUG:teuthology.orchestra.run.trial115:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-22T23:27:18.167 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-22T23:27:18.232 DEBUG:teuthology.orchestra.run.trial115:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-22T23:27:18.282 DEBUG:teuthology.orchestra.run.trial115:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2026-02-22T23:27:18.330 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-22T23:27:18.396 DEBUG:teuthology.orchestra.run.trial115:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2026-02-22T23:27:18.447 DEBUG:teuthology.orchestra.run.trial115:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2026-02-22T23:27:18.494 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-22T23:27:18.561 DEBUG:teuthology.orchestra.run.trial115:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2026-02-22T23:27:18.611 DEBUG:teuthology.orchestra.run.trial115:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-22T23:27:18.659 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-22T23:27:18.659 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-22T23:27:18.660 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-22T23:27:18.701 INFO:tasks.ceph.mon.a:Started 2026-02-22T23:27:18.701 INFO:tasks.ceph.mon.b:Restarting daemon 2026-02-22T23:27:18.701 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2026-02-22T23:27:18.703 INFO:tasks.ceph.mon.b:Started 2026-02-22T23:27:18.703 INFO:tasks.ceph.mon.c:Restarting daemon 2026-02-22T23:27:18.703 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2026-02-22T23:27:18.705 INFO:tasks.ceph.mon.c:Started 2026-02-22T23:27:18.705 DEBUG:teuthology.orchestra.run.trial115:> sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force 2026-02-22T23:27:24.126 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-22T23:27:24.127 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-22T23:27:24.127 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-22T23:27:24.129 INFO:tasks.ceph.mgr.x:Started 2026-02-22T23:27:24.130 INFO:tasks.ceph:Setting crush tunables to default 2026-02-22T23:27:24.130 DEBUG:teuthology.orchestra.run.trial115:> sudo ceph --cluster ceph osd crush tunables default 2026-02-22T23:27:24.223 INFO:teuthology.orchestra.run.trial115.stderr:adjusted tunables profile to default 2026-02-22T23:27:24.233 INFO:tasks.ceph:check_enable_crimson: False 2026-02-22T23:27:24.233 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-22T23:27:24.233 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:27:24.233 DEBUG:teuthology.orchestra.run.trial115:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-22T23:27:24.237 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:24.234+0000 7f28568c3280 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2026-02-22T23:27:24.242 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:27:24.242 DEBUG:teuthology.orchestra.run.trial115:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-22T23:27:24.264 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:24.262+0000 7f28568c3280 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2026-02-22T23:27:24.272 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:27:24.272 DEBUG:teuthology.orchestra.run.trial115:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-22T23:27:24.303 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:24.298+0000 7f28568c3280 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2026-02-22T23:27:24.310 DEBUG:teuthology.orchestra.run.trial115:> set -ex 2026-02-22T23:27:24.310 DEBUG:teuthology.orchestra.run.trial115:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2026-02-22T23:27:24.358 DEBUG:teuthology.orchestra.run.trial115:> sudo ceph --cluster ceph osd new bffc6a3f-193a-457c-aedc-72aed6eb39e5 0 2026-02-22T23:27:24.375 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:24.370+0000 7f28568c3280 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2026-02-22T23:27:24.457 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:24.454+0000 7f28568c3280 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2026-02-22T23:27:24.472 INFO:teuthology.orchestra.run.trial115.stdout:0 2026-02-22T23:27:24.481 DEBUG:teuthology.orchestra.run.trial115:> sudo ceph --cluster ceph osd new 5b969a7a-bb01-4c32-84bc-ca4ac4d91e33 1 2026-02-22T23:27:24.496 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:24.494+0000 7f28568c3280 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2026-02-22T23:27:24.574 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:24.570+0000 7f28568c3280 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2026-02-22T23:27:24.578 INFO:teuthology.orchestra.run.trial115.stdout:1 2026-02-22T23:27:24.587 DEBUG:teuthology.orchestra.run.trial115:> sudo ceph --cluster ceph osd new 6da393ab-bd56-4d92-a2e8-9807271ac6e4 2 2026-02-22T23:27:24.724 INFO:teuthology.orchestra.run.trial115.stdout:2 2026-02-22T23:27:24.734 DEBUG:teuthology.orchestra.run.trial115:> sudo ceph --cluster ceph osd new 288ba70e-ec1b-4b0e-8609-cafdcbba9bda 3 2026-02-22T23:27:24.755 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:24.750+0000 7f28568c3280 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2026-02-22T23:27:24.829 INFO:teuthology.orchestra.run.trial115.stdout:3 2026-02-22T23:27:24.839 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-22T23:27:24.840 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-02-22T23:27:24.885 INFO:tasks.ceph.osd.0:Started 2026-02-22T23:27:24.885 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-22T23:27:24.885 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-02-22T23:27:24.887 INFO:tasks.ceph.osd.1:Started 2026-02-22T23:27:24.888 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-22T23:27:24.888 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-02-22T23:27:24.890 INFO:tasks.ceph.osd.2:Started 2026-02-22T23:27:24.890 INFO:tasks.ceph.osd.3:Restarting daemon 2026-02-22T23:27:24.890 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2026-02-22T23:27:24.892 INFO:tasks.ceph.osd.3:Started 2026-02-22T23:27:24.893 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T23:27:24.985 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:24.986 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":6,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:27:24.830980+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T23:27:24.830980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:27:24.997 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-22T23:27:24.997 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-22T23:27:25.060 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:25.058+0000 7f28568c3280 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2026-02-22T23:27:25.152 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:25.150+0000 7f28568c3280 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2026-02-22T23:27:25.179 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:25.174+0000 7f28568c3280 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2026-02-22T23:27:25.210 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:25.206+0000 7f28568c3280 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2026-02-22T23:27:25.252 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:25.250+0000 7f28568c3280 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2026-02-22T23:27:25.351 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:25.346+0000 7f28568c3280 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2026-02-22T23:27:25.401 DEBUG:teuthology.orchestra.run.trial115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T23:27:25.487 INFO:teuthology.misc.health.trial115.stdout: 2026-02-22T23:27:25.488 INFO:teuthology.misc.health.trial115.stdout:{"epoch":6,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:27:24.830980+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T23:27:24.830980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:27:25.496 DEBUG:teuthology.misc:0 of 4 OSDs are up 2026-02-22T23:27:25.541 INFO:tasks.ceph.mgr.x.trial115.stderr:ERROR:root:Module 'xmltodict' is not installed. 2026-02-22T23:27:25.714 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:25.710+0000 7f28568c3280 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2026-02-22T23:27:25.746 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:25.742+0000 7f28568c3280 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2026-02-22T23:27:25.838 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:25.834+0000 7f28568c3280 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2026-02-22T23:27:25.863 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:25.858+0000 7f28568c3280 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2026-02-22T23:27:25.889 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:25.886+0000 7f28568c3280 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2026-02-22T23:27:25.916 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:25.914+0000 7f28568c3280 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2026-02-22T23:27:26.036 INFO:tasks.ceph.mgr.x.trial115.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-22T23:27:26.036 INFO:tasks.ceph.mgr.x.trial115.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-22T23:27:26.036 INFO:tasks.ceph.mgr.x.trial115.stderr: from numpy import show_config as show_numpy_config 2026-02-22T23:27:26.043 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:26.038+0000 7f28568c3280 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2026-02-22T23:27:26.069 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:26.066+0000 7f28568c3280 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2026-02-22T23:27:26.108 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:26.106+0000 7f28568c3280 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2026-02-22T23:27:26.429 INFO:tasks.ceph.osd.2.trial115.stderr:2026-02-22T23:27:26.426+0000 7faca6cfd8c0 -1 Falling back to public interface 2026-02-22T23:27:26.717 INFO:tasks.ceph.osd.1.trial115.stderr:2026-02-22T23:27:26.714+0000 7fe3523cf8c0 -1 Falling back to public interface 2026-02-22T23:27:26.937 INFO:tasks.ceph.osd.0.trial115.stderr:2026-02-22T23:27:26.934+0000 7f1d0a1698c0 -1 Falling back to public interface 2026-02-22T23:27:26.944 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:27:26.942+0000 7f4d41c4d8c0 -1 Falling back to public interface 2026-02-22T23:27:27.137 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:27.134+0000 7f2775311640 -1 mgr.server handle_report got status from non-daemon mon.c 2026-02-22T23:27:27.137 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:27:27.134+0000 7f2775311640 -1 mgr.server handle_report got status from non-daemon mon.b 2026-02-22T23:27:30.547 INFO:tasks.ceph.osd.2.trial115.stderr:2026-02-22T23:27:30.542+0000 7faca6cfd8c0 -1 osd.2 0 log_to_monitors true 2026-02-22T23:27:30.634 INFO:tasks.ceph.osd.1.trial115.stderr:2026-02-22T23:27:30.630+0000 7fe3523cf8c0 -1 osd.1 0 log_to_monitors true 2026-02-22T23:27:31.061 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:27:31.058+0000 7f4d41c4d8c0 -1 osd.3 0 log_to_monitors true 2026-02-22T23:27:31.271 INFO:tasks.ceph.osd.0.trial115.stderr:2026-02-22T23:27:31.266+0000 7f1d0a1698c0 -1 osd.0 0 log_to_monitors true 2026-02-22T23:27:31.899 DEBUG:teuthology.orchestra.run.trial115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T23:27:32.039 INFO:teuthology.misc.health.trial115.stdout: 2026-02-22T23:27:32.039 INFO:teuthology.misc.health.trial115.stdout:{"epoch":7,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:27:31.138686+0000","last_up_change":"0.000000","last_in_change":"2026-02-22T23:27:24.830980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:27:32.048 DEBUG:teuthology.misc:0 of 4 OSDs are up 2026-02-22T23:27:32.143 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:27:32.138+0000 7f4d3dbf6640 -1 osd.3 0 waiting for initial osdmap 2026-02-22T23:27:32.143 INFO:tasks.ceph.osd.2.trial115.stderr:2026-02-22T23:27:32.138+0000 7faca2ca6640 -1 osd.2 0 waiting for initial osdmap 2026-02-22T23:27:32.144 INFO:tasks.ceph.osd.1.trial115.stderr:2026-02-22T23:27:32.142+0000 7fe34e378640 -1 osd.1 0 waiting for initial osdmap 2026-02-22T23:27:32.146 INFO:tasks.ceph.osd.2.trial115.stderr:2026-02-22T23:27:32.142+0000 7fac9aad9640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T23:27:32.146 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:27:32.142+0000 7f4d35a29640 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T23:27:32.147 INFO:tasks.ceph.osd.1.trial115.stderr:2026-02-22T23:27:32.142+0000 7fe3461ab640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T23:27:33.146 INFO:tasks.ceph.osd.0.trial115.stderr:2026-02-22T23:27:33.142+0000 7f1d06112640 -1 osd.0 0 waiting for initial osdmap 2026-02-22T23:27:33.149 INFO:tasks.ceph.osd.0.trial115.stderr:2026-02-22T23:27:33.146+0000 7f1d01f4d640 -1 osd.0 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T23:27:38.451 DEBUG:teuthology.orchestra.run.trial115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-22T23:27:38.588 INFO:teuthology.misc.health.trial115.stdout: 2026-02-22T23:27:38.588 INFO:teuthology.misc.health.trial115.stdout:{"epoch":13,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:27:37.156520+0000","last_up_change":"2026-02-22T23:27:34.146668+0000","last_in_change":"2026-02-22T23:27:24.830980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T23:27:34.171905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6809","nonce":3878013816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6810","nonce":3878013816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6812","nonce":3878013816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6811","nonce":3878013816}]},"public_addr":"10.20.193.115:6809/3878013816","cluster_addr":"10.20.193.115:6810/3878013816","heartbeat_back_addr":"10.20.193.115:6812/3878013816","heartbeat_front_addr":"10.20.193.115:6811/3878013816","state":["exists","up"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6805","nonce":464259751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6806","nonce":464259751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6808","nonce":464259751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6807","nonce":464259751}]},"public_addr":"10.20.193.115:6805/464259751","cluster_addr":"10.20.193.115:6806/464259751","heartbeat_back_addr":"10.20.193.115:6808/464259751","heartbeat_front_addr":"10.20.193.115:6807/464259751","state":["exists","up"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6801","nonce":1978117969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6802","nonce":1978117969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6804","nonce":1978117969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6803","nonce":1978117969}]},"public_addr":"10.20.193.115:6801/1978117969","cluster_addr":"10.20.193.115:6802/1978117969","heartbeat_back_addr":"10.20.193.115:6804/1978117969","heartbeat_front_addr":"10.20.193.115:6803/1978117969","state":["exists","up"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6813","nonce":3651482031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6814","nonce":3651482031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6816","nonce":3651482031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6815","nonce":3651482031}]},"public_addr":"10.20.193.115:6813/3651482031","cluster_addr":"10.20.193.115:6814/3651482031","heartbeat_back_addr":"10.20.193.115:6816/3651482031","heartbeat_front_addr":"10.20.193.115:6815/3651482031","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:27:38.599 DEBUG:teuthology.misc:4 of 4 OSDs are up 2026-02-22T23:27:38.599 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-22T23:27:38.600 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-22T23:27:38.600 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-22T23:27:38.600 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-22T23:27:38.601 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-22T23:27:38.775 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:38.786 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":4,"flags":0,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6800","nonce":2458122209}]},"active_addr":"10.20.193.115:6800/2458122209","active_change":"2026-02-22T23:27:26.130975+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number. if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.115:0","nonce":589057061}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.115:0","nonce":1262257434}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.115:0","nonce":4010613344}]}]} 2026-02-22T23:27:38.790 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-22T23:27:38.790 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-22T23:27:38.790 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T23:27:38.934 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:38.934 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":13,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:27:37.156520+0000","last_up_change":"2026-02-22T23:27:34.146668+0000","last_in_change":"2026-02-22T23:27:24.830980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T23:27:34.171905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6809","nonce":3878013816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6810","nonce":3878013816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6812","nonce":3878013816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6811","nonce":3878013816}]},"public_addr":"10.20.193.115:6809/3878013816","cluster_addr":"10.20.193.115:6810/3878013816","heartbeat_back_addr":"10.20.193.115:6812/3878013816","heartbeat_front_addr":"10.20.193.115:6811/3878013816","state":["exists","up"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6805","nonce":464259751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6806","nonce":464259751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6808","nonce":464259751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6807","nonce":464259751}]},"public_addr":"10.20.193.115:6805/464259751","cluster_addr":"10.20.193.115:6806/464259751","heartbeat_back_addr":"10.20.193.115:6808/464259751","heartbeat_front_addr":"10.20.193.115:6807/464259751","state":["exists","up"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6801","nonce":1978117969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6802","nonce":1978117969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6804","nonce":1978117969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6803","nonce":1978117969}]},"public_addr":"10.20.193.115:6801/1978117969","cluster_addr":"10.20.193.115:6802/1978117969","heartbeat_back_addr":"10.20.193.115:6804/1978117969","heartbeat_front_addr":"10.20.193.115:6803/1978117969","state":["exists","up"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6813","nonce":3651482031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6814","nonce":3651482031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6816","nonce":3651482031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6815","nonce":3651482031}]},"public_addr":"10.20.193.115:6813/3651482031","cluster_addr":"10.20.193.115:6814/3651482031","heartbeat_back_addr":"10.20.193.115:6816/3651482031","heartbeat_front_addr":"10.20.193.115:6815/3651482031","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:27:38.947 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-22T23:27:38.947 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T23:27:39.091 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:39.091 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":13,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:27:37.156520+0000","last_up_change":"2026-02-22T23:27:34.146668+0000","last_in_change":"2026-02-22T23:27:24.830980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T23:27:34.171905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6809","nonce":3878013816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6810","nonce":3878013816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6812","nonce":3878013816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6811","nonce":3878013816}]},"public_addr":"10.20.193.115:6809/3878013816","cluster_addr":"10.20.193.115:6810/3878013816","heartbeat_back_addr":"10.20.193.115:6812/3878013816","heartbeat_front_addr":"10.20.193.115:6811/3878013816","state":["exists","up"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6805","nonce":464259751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6806","nonce":464259751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6808","nonce":464259751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6807","nonce":464259751}]},"public_addr":"10.20.193.115:6805/464259751","cluster_addr":"10.20.193.115:6806/464259751","heartbeat_back_addr":"10.20.193.115:6808/464259751","heartbeat_front_addr":"10.20.193.115:6807/464259751","state":["exists","up"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6801","nonce":1978117969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6802","nonce":1978117969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6804","nonce":1978117969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6803","nonce":1978117969}]},"public_addr":"10.20.193.115:6801/1978117969","cluster_addr":"10.20.193.115:6802/1978117969","heartbeat_back_addr":"10.20.193.115:6804/1978117969","heartbeat_front_addr":"10.20.193.115:6803/1978117969","state":["exists","up"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6813","nonce":3651482031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6814","nonce":3651482031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6816","nonce":3651482031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6815","nonce":3651482031}]},"public_addr":"10.20.193.115:6813/3651482031","cluster_addr":"10.20.193.115:6814/3651482031","heartbeat_back_addr":"10.20.193.115:6816/3651482031","heartbeat_front_addr":"10.20.193.115:6815/3651482031","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:27:39.104 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:27:39.104 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:27:39.105 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:27:39.105 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:27:39.191 INFO:teuthology.orchestra.run.trial115.stdout:38654705667 2026-02-22T23:27:39.192 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:27:39.193 INFO:teuthology.orchestra.run.trial115.stdout:38654705667 2026-02-22T23:27:39.194 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:27:39.195 INFO:teuthology.orchestra.run.trial115.stdout:38654705667 2026-02-22T23:27:39.195 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:27:39.200 INFO:teuthology.orchestra.run.trial115.stdout:42949672962 2026-02-22T23:27:39.201 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:27:39.350 INFO:teuthology.orchestra.run.trial115.stdout:38654705665 2026-02-22T23:27:39.362 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705665 for osd.1 2026-02-22T23:27:39.387 INFO:teuthology.orchestra.run.trial115.stdout:38654705665 2026-02-22T23:27:39.387 INFO:teuthology.orchestra.run.trial115.stdout:38654705665 2026-02-22T23:27:39.393 INFO:teuthology.orchestra.run.trial115.stdout:42949672961 2026-02-22T23:27:39.399 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705665 for osd.3 2026-02-22T23:27:39.400 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705665 for osd.2 2026-02-22T23:27:39.405 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672962 got 42949672961 for osd.0 2026-02-22T23:27:40.363 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:27:40.399 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:27:40.401 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:27:40.406 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:27:40.518 INFO:teuthology.orchestra.run.trial115.stdout:38654705667 2026-02-22T23:27:40.530 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.1 2026-02-22T23:27:40.531 DEBUG:teuthology.parallel:result is None 2026-02-22T23:27:40.552 INFO:teuthology.orchestra.run.trial115.stdout:42949672962 2026-02-22T23:27:40.553 INFO:teuthology.orchestra.run.trial115.stdout:38654705667 2026-02-22T23:27:40.553 INFO:teuthology.orchestra.run.trial115.stdout:38654705667 2026-02-22T23:27:40.564 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672962 got 42949672962 for osd.0 2026-02-22T23:27:40.564 DEBUG:teuthology.parallel:result is None 2026-02-22T23:27:40.565 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.2 2026-02-22T23:27:40.566 DEBUG:teuthology.parallel:result is None 2026-02-22T23:27:40.567 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.3 2026-02-22T23:27:40.567 DEBUG:teuthology.parallel:result is None 2026-02-22T23:27:40.567 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-22T23:27:40.567 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:27:40.753 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:40.753 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:27:40.765 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2026-02-22T23:27:40.135931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":2,"kb":377487360,"kb_used":108352,"kb_used_data":496,"kb_used_omap":6,"kb_used_meta":107769,"kb_avail":377379008,"statfs":{"total":386547056640,"available":386436104192,"internally_reserved":0,"allocated":507904,"data_stored":768432,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":6374,"internal_metadata":110356250},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"12'17","reported_seq":35,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T23:27:37.160668+0000","last_change":"2026-02-22T23:27:36.327993+0000","last_active":"2026-02-22T23:27:37.160668+0000","last_peered":"2026-02-22T23:27:37.160668+0000","last_clean":"2026-02-22T23:27:37.160668+0000","last_became_active":"2026-02-22T23:27:36.327168+0000","last_became_peered":"2026-02-22T23:27:36.327168+0000","last_unstale":"2026-02-22T23:27:37.160668+0000","last_undegraded":"2026-02-22T23:27:37.160668+0000","last_fullsized":"2026-02-22T23:27:37.160668+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:19:08.163773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949672962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27100,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344740,"statfs":{"total":96636764160,"available":96609013760,"internally_reserved":0,"allocated":139264,"data_stored":356204,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27076,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344764,"statfs":{"total":96636764160,"available":96609038336,"internally_reserved":0,"allocated":114688,"data_stored":28012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27589064},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27076,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344764,"statfs":{"total":96636764160,"available":96609038336,"internally_reserved":0,"allocated":114688,"data_stored":28012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27100,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344740,"statfs":{"total":96636764160,"available":96609013760,"internally_reserved":0,"allocated":139264,"data_stored":356204,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T23:27:40.766 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:27:40.919 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:40.919 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:27:40.929 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2026-02-22T23:27:40.135931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":2,"kb":377487360,"kb_used":108352,"kb_used_data":496,"kb_used_omap":6,"kb_used_meta":107769,"kb_avail":377379008,"statfs":{"total":386547056640,"available":386436104192,"internally_reserved":0,"allocated":507904,"data_stored":768432,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":6374,"internal_metadata":110356250},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"12'17","reported_seq":35,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-22T23:27:37.160668+0000","last_change":"2026-02-22T23:27:36.327993+0000","last_active":"2026-02-22T23:27:37.160668+0000","last_peered":"2026-02-22T23:27:37.160668+0000","last_clean":"2026-02-22T23:27:37.160668+0000","last_became_active":"2026-02-22T23:27:36.327168+0000","last_became_peered":"2026-02-22T23:27:36.327168+0000","last_unstale":"2026-02-22T23:27:37.160668+0000","last_undegraded":"2026-02-22T23:27:37.160668+0000","last_fullsized":"2026-02-22T23:27:37.160668+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:19:08.163773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949672962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27100,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344740,"statfs":{"total":96636764160,"available":96609013760,"internally_reserved":0,"allocated":139264,"data_stored":356204,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27076,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344764,"statfs":{"total":96636764160,"available":96609038336,"internally_reserved":0,"allocated":114688,"data_stored":28012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27589064},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27076,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344764,"statfs":{"total":96636764160,"available":96609038336,"internally_reserved":0,"allocated":114688,"data_stored":28012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27100,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344740,"statfs":{"total":96636764160,"available":96609013760,"internally_reserved":0,"allocated":139264,"data_stored":356204,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T23:27:40.930 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-22T23:27:40.930 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-22T23:27:40.930 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-22T23:27:40.931 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-22T23:27:41.090 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:41.090 INFO:teuthology.orchestra.run.trial115.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-22T23:27:41.102 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-22T23:27:41.102 INFO:teuthology.run_tasks:Running task ec_inconsistent_hinfo... 2026-02-22T23:27:41.118 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T23:27:41.263 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:41.264 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":13,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:27:37.156520+0000","last_up_change":"2026-02-22T23:27:34.146668+0000","last_in_change":"2026-02-22T23:27:24.830980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T23:27:34.171905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6809","nonce":3878013816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6810","nonce":3878013816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6812","nonce":3878013816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6811","nonce":3878013816}]},"public_addr":"10.20.193.115:6809/3878013816","cluster_addr":"10.20.193.115:6810/3878013816","heartbeat_back_addr":"10.20.193.115:6812/3878013816","heartbeat_front_addr":"10.20.193.115:6811/3878013816","state":["exists","up"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6805","nonce":464259751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6806","nonce":464259751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6808","nonce":464259751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6807","nonce":464259751}]},"public_addr":"10.20.193.115:6805/464259751","cluster_addr":"10.20.193.115:6806/464259751","heartbeat_back_addr":"10.20.193.115:6808/464259751","heartbeat_front_addr":"10.20.193.115:6807/464259751","state":["exists","up"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6801","nonce":1978117969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6802","nonce":1978117969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6804","nonce":1978117969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6803","nonce":1978117969}]},"public_addr":"10.20.193.115:6801/1978117969","cluster_addr":"10.20.193.115:6802/1978117969","heartbeat_back_addr":"10.20.193.115:6804/1978117969","heartbeat_front_addr":"10.20.193.115:6803/1978117969","state":["exists","up"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6813","nonce":3651482031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6814","nonce":3651482031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6816","nonce":3651482031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6815","nonce":3651482031}]},"public_addr":"10.20.193.115:6813/3651482031","cluster_addr":"10.20.193.115:6814/3651482031","heartbeat_back_addr":"10.20.193.115:6816/3651482031","heartbeat_front_addr":"10.20.193.115:6815/3651482031","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:27:41.274 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2026-02-22T23:27:34.171905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '13', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4, 'score_stable': 4, 'optimal_score': 0.5, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2026-02-22T23:27:41.274 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2026-02-22T23:27:41.420 INFO:teuthology.orchestra.run.trial115.stdout:pg_num: 1 2026-02-22T23:27:41.431 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set backfill_unfound k=2 m=1 crush-failure-domain=osd 2026-02-22T23:27:42.181 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:creating pool_name unique_pool_0 2026-02-22T23:27:42.182 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 1 1 erasure backfill_unfound 2026-02-22T23:27:44.176 INFO:teuthology.orchestra.run.trial115.stderr:pool 'unique_pool_0' created 2026-02-22T23:27:44.189 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 min_size 2 2026-02-22T23:27:45.180 INFO:teuthology.orchestra.run.trial115.stderr:set pool 2 min_size to 2 2026-02-22T23:27:45.192 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2026-02-22T23:27:46.183 INFO:teuthology.orchestra.run.trial115.stderr:enabled application 'rados' on pool 'unique_pool_0' 2026-02-22T23:27:47.196 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_autoscale_mode off 2026-02-22T23:27:48.189 INFO:teuthology.orchestra.run.trial115.stderr:set pool 2 pg_autoscale_mode to off 2026-02-22T23:27:48.201 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:27:48.202 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:27:48.202 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:27:48.202 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:27:48.287 INFO:teuthology.orchestra.run.trial115.stdout:42949672965 2026-02-22T23:27:48.288 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:27:48.289 INFO:teuthology.orchestra.run.trial115.stdout:38654705670 2026-02-22T23:27:48.289 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:27:48.292 INFO:teuthology.orchestra.run.trial115.stdout:38654705669 2026-02-22T23:27:48.292 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:27:48.293 INFO:teuthology.orchestra.run.trial115.stdout:38654705669 2026-02-22T23:27:48.294 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:27:48.485 INFO:teuthology.orchestra.run.trial115.stdout:42949672964 2026-02-22T23:27:48.486 INFO:teuthology.orchestra.run.trial115.stdout:38654705668 2026-02-22T23:27:48.497 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672965 got 42949672964 for osd.0 2026-02-22T23:27:48.498 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705670 got 38654705668 for osd.1 2026-02-22T23:27:48.527 INFO:teuthology.orchestra.run.trial115.stdout:38654705668 2026-02-22T23:27:48.530 INFO:teuthology.orchestra.run.trial115.stdout:38654705668 2026-02-22T23:27:48.538 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705669 got 38654705668 for osd.3 2026-02-22T23:27:48.542 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705669 got 38654705668 for osd.2 2026-02-22T23:27:49.498 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:27:49.499 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:27:49.538 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:27:49.543 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:27:49.647 INFO:teuthology.orchestra.run.trial115.stdout:38654705668 2026-02-22T23:27:49.660 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705670 got 38654705668 for osd.1 2026-02-22T23:27:49.668 INFO:teuthology.orchestra.run.trial115.stdout:42949672964 2026-02-22T23:27:49.681 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672965 got 42949672964 for osd.0 2026-02-22T23:27:49.692 INFO:teuthology.orchestra.run.trial115.stdout:38654705668 2026-02-22T23:27:49.705 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705669 got 38654705668 for osd.2 2026-02-22T23:27:49.706 INFO:teuthology.orchestra.run.trial115.stdout:38654705668 2026-02-22T23:27:49.718 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705669 got 38654705668 for osd.3 2026-02-22T23:27:50.660 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:27:50.682 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:27:50.706 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:27:50.719 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:27:50.810 INFO:teuthology.orchestra.run.trial115.stdout:38654705670 2026-02-22T23:27:50.823 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705670 got 38654705670 for osd.1 2026-02-22T23:27:50.823 DEBUG:teuthology.parallel:result is None 2026-02-22T23:27:50.848 INFO:teuthology.orchestra.run.trial115.stdout:42949672965 2026-02-22T23:27:50.860 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672965 got 42949672965 for osd.0 2026-02-22T23:27:50.860 DEBUG:teuthology.parallel:result is None 2026-02-22T23:27:50.890 INFO:teuthology.orchestra.run.trial115.stdout:38654705670 2026-02-22T23:27:50.902 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705669 got 38654705670 for osd.2 2026-02-22T23:27:50.902 DEBUG:teuthology.parallel:result is None 2026-02-22T23:27:50.902 INFO:teuthology.orchestra.run.trial115.stdout:38654705670 2026-02-22T23:27:50.914 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705669 got 38654705670 for osd.3 2026-02-22T23:27:50.914 DEBUG:teuthology.parallel:result is None 2026-02-22T23:27:50.914 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for clean 2026-02-22T23:27:50.914 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:27:51.105 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:51.106 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:27:51.117 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2026-02-22T23:27:50.137365+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108524,"kb_used_data":664,"kb_used_omap":6,"kb_used_meta":107769,"kb_avail":377378836,"statfs":{"total":386547056640,"available":386435928064,"internally_reserved":0,"allocated":679936,"data_stored":846011,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":6374,"internal_metadata":110356250},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.960468"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-22T23:27:48.191187+0000","last_change":"2026-02-22T23:27:45.185356+0000","last_active":"2026-02-22T23:27:48.191187+0000","last_peered":"2026-02-22T23:27:48.191187+0000","last_clean":"2026-02-22T23:27:48.191187+0000","last_became_active":"2026-02-22T23:27:45.185140+0000","last_became_peered":"2026-02-22T23:27:45.185140+0000","last_unstale":"2026-02-22T23:27:48.191187+0000","last_undegraded":"2026-02-22T23:27:48.191187+0000","last_fullsized":"2026-02-22T23:27:48.191187+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:44.173266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:44.173266+0000","last_clean_scrub_stamp":"2026-02-22T23:27:44.173266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:59:21.383326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":47,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-22T23:27:48.190934+0000","last_change":"2026-02-22T23:27:36.327993+0000","last_active":"2026-02-22T23:27:48.190934+0000","last_peered":"2026-02-22T23:27:48.190934+0000","last_clean":"2026-02-22T23:27:48.190934+0000","last_became_active":"2026-02-22T23:27:36.327168+0000","last_became_peered":"2026-02-22T23:27:36.327168+0000","last_unstale":"2026-02-22T23:27:48.190934+0000","last_undegraded":"2026-02-22T23:27:48.190934+0000","last_fullsized":"2026-02-22T23:27:48.190934+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:19:08.163773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949672965,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":180224,"data_stored":374652,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27128,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344712,"statfs":{"total":96636764160,"available":96608985088,"internally_reserved":0,"allocated":163840,"data_stored":50247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27589064},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27116,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":155648,"data_stored":46460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":180224,"data_stored":374652,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T23:27:51.118 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:27:51.271 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:51.272 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:27:51.282 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2026-02-22T23:27:50.137365+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108524,"kb_used_data":664,"kb_used_omap":6,"kb_used_meta":107769,"kb_avail":377378836,"statfs":{"total":386547056640,"available":386435928064,"internally_reserved":0,"allocated":679936,"data_stored":846011,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":6374,"internal_metadata":110356250},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.960468"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-22T23:27:48.191187+0000","last_change":"2026-02-22T23:27:45.185356+0000","last_active":"2026-02-22T23:27:48.191187+0000","last_peered":"2026-02-22T23:27:48.191187+0000","last_clean":"2026-02-22T23:27:48.191187+0000","last_became_active":"2026-02-22T23:27:45.185140+0000","last_became_peered":"2026-02-22T23:27:45.185140+0000","last_unstale":"2026-02-22T23:27:48.191187+0000","last_undegraded":"2026-02-22T23:27:48.191187+0000","last_fullsized":"2026-02-22T23:27:48.191187+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:44.173266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:44.173266+0000","last_clean_scrub_stamp":"2026-02-22T23:27:44.173266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:59:21.383326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":47,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-22T23:27:48.190934+0000","last_change":"2026-02-22T23:27:36.327993+0000","last_active":"2026-02-22T23:27:48.190934+0000","last_peered":"2026-02-22T23:27:48.190934+0000","last_clean":"2026-02-22T23:27:48.190934+0000","last_became_active":"2026-02-22T23:27:36.327168+0000","last_became_peered":"2026-02-22T23:27:36.327168+0000","last_unstale":"2026-02-22T23:27:48.190934+0000","last_undegraded":"2026-02-22T23:27:48.190934+0000","last_fullsized":"2026-02-22T23:27:48.190934+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:19:08.163773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949672965,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":180224,"data_stored":374652,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27128,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344712,"statfs":{"total":96636764160,"available":96608985088,"internally_reserved":0,"allocated":163840,"data_stored":50247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27589064},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27116,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":155648,"data_stored":46460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":180224,"data_stored":374652,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T23:27:51.283 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:clean! 2026-02-22T23:27:51.283 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T23:27:51.428 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:51.428 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":19,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:27:48.184904+0000","last_up_change":"2026-02-22T23:27:34.146668+0000","last_in_change":"2026-02-22T23:27:24.830980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T23:27:34.171905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-22T23:27:43.179120+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6809","nonce":3878013816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6810","nonce":3878013816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6812","nonce":3878013816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6811","nonce":3878013816}]},"public_addr":"10.20.193.115:6809/3878013816","cluster_addr":"10.20.193.115:6810/3878013816","heartbeat_back_addr":"10.20.193.115:6812/3878013816","heartbeat_front_addr":"10.20.193.115:6811/3878013816","state":["exists","up"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6805","nonce":464259751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6806","nonce":464259751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6808","nonce":464259751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6807","nonce":464259751}]},"public_addr":"10.20.193.115:6805/464259751","cluster_addr":"10.20.193.115:6806/464259751","heartbeat_back_addr":"10.20.193.115:6808/464259751","heartbeat_front_addr":"10.20.193.115:6807/464259751","state":["exists","up"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6801","nonce":1978117969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6802","nonce":1978117969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6804","nonce":1978117969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6803","nonce":1978117969}]},"public_addr":"10.20.193.115:6801/1978117969","cluster_addr":"10.20.193.115:6802/1978117969","heartbeat_back_addr":"10.20.193.115:6804/1978117969","heartbeat_front_addr":"10.20.193.115:6803/1978117969","state":["exists","up"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6813","nonce":3651482031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6814","nonce":3651482031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6816","nonce":3651482031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6815","nonce":3651482031}]},"public_addr":"10.20.193.115:6813/3651482031","cluster_addr":"10.20.193.115:6814/3651482031","heartbeat_back_addr":"10.20.193.115:6816/3651482031","heartbeat_front_addr":"10.20.193.115:6815/3651482031","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:27:51.439 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:27:51.583 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:51.584 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:27:51.593 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2026-02-22T23:27:50.137365+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108524,"kb_used_data":664,"kb_used_omap":6,"kb_used_meta":107769,"kb_avail":377378836,"statfs":{"total":386547056640,"available":386435928064,"internally_reserved":0,"allocated":679936,"data_stored":846011,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":6374,"internal_metadata":110356250},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.960468"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-22T23:27:48.191187+0000","last_change":"2026-02-22T23:27:45.185356+0000","last_active":"2026-02-22T23:27:48.191187+0000","last_peered":"2026-02-22T23:27:48.191187+0000","last_clean":"2026-02-22T23:27:48.191187+0000","last_became_active":"2026-02-22T23:27:45.185140+0000","last_became_peered":"2026-02-22T23:27:45.185140+0000","last_unstale":"2026-02-22T23:27:48.191187+0000","last_undegraded":"2026-02-22T23:27:48.191187+0000","last_fullsized":"2026-02-22T23:27:48.191187+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:44.173266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:44.173266+0000","last_clean_scrub_stamp":"2026-02-22T23:27:44.173266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:59:21.383326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":47,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-22T23:27:48.190934+0000","last_change":"2026-02-22T23:27:36.327993+0000","last_active":"2026-02-22T23:27:48.190934+0000","last_peered":"2026-02-22T23:27:48.190934+0000","last_clean":"2026-02-22T23:27:48.190934+0000","last_became_active":"2026-02-22T23:27:36.327168+0000","last_became_peered":"2026-02-22T23:27:36.327168+0000","last_unstale":"2026-02-22T23:27:48.190934+0000","last_undegraded":"2026-02-22T23:27:48.190934+0000","last_fullsized":"2026-02-22T23:27:48.190934+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:19:08.163773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949672965,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":180224,"data_stored":374652,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27128,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344712,"statfs":{"total":96636764160,"available":96608985088,"internally_reserved":0,"allocated":163840,"data_stored":50247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27589064},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27116,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":155648,"data_stored":46460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":180224,"data_stored":374652,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T23:27:51.594 INFO:tasks.ec_inconsistent_hinfo:acting=[3, 1, 2] 2026-02-22T23:27:51.594 INFO:tasks.util.rados:rados -p unique_pool_0 put dummy /etc/group 2026-02-22T23:27:51.595 DEBUG:teuthology.orchestra.run.trial115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put dummy /etc/group 2026-02-22T23:27:51.623 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:27:51.624 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:27:51.746 INFO:teuthology.orchestra.run.trial115.stdout:38654705671 2026-02-22T23:27:51.747 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:27:51.748 INFO:teuthology.orchestra.run.trial115.stdout:42949672967 2026-02-22T23:27:51.748 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:27:51.931 INFO:teuthology.orchestra.run.trial115.stdout:42949672965 2026-02-22T23:27:51.932 INFO:teuthology.orchestra.run.trial115.stdout:38654705670 2026-02-22T23:27:51.944 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705671 got 38654705670 for osd.1 2026-02-22T23:27:51.945 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672967 got 42949672965 for osd.0 2026-02-22T23:27:52.945 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:27:52.946 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:27:53.093 INFO:teuthology.orchestra.run.trial115.stdout:38654705671 2026-02-22T23:27:53.095 INFO:teuthology.orchestra.run.trial115.stdout:42949672967 2026-02-22T23:27:53.104 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705671 got 38654705671 for osd.1 2026-02-22T23:27:53.104 DEBUG:teuthology.parallel:result is None 2026-02-22T23:27:53.106 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672967 got 42949672967 for osd.0 2026-02-22T23:27:53.106 DEBUG:teuthology.parallel:result is None 2026-02-22T23:27:53.106 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for recovery to complete 2026-02-22T23:27:53.107 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:27:53.295 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:53.295 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:27:53.305 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2026-02-22T23:27:52.137589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108544,"kb_used_data":684,"kb_used_omap":6,"kb_used_meta":107769,"kb_avail":377378816,"statfs":{"total":386547056640,"available":386435907584,"internally_reserved":0,"allocated":700416,"data_stored":861777,"data_compressed":3317,"data_compressed_allocated":45056,"data_compressed_original":663552,"omap_allocated":6374,"internal_metadata":110356250},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.956362"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-22T23:27:48.191187+0000","last_change":"2026-02-22T23:27:45.185356+0000","last_active":"2026-02-22T23:27:48.191187+0000","last_peered":"2026-02-22T23:27:48.191187+0000","last_clean":"2026-02-22T23:27:48.191187+0000","last_became_active":"2026-02-22T23:27:45.185140+0000","last_became_peered":"2026-02-22T23:27:45.185140+0000","last_unstale":"2026-02-22T23:27:48.191187+0000","last_undegraded":"2026-02-22T23:27:48.191187+0000","last_fullsized":"2026-02-22T23:27:48.191187+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:44.173266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:44.173266+0000","last_clean_scrub_stamp":"2026-02-22T23:27:44.173266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:59:21.383326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":47,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-22T23:27:48.190934+0000","last_change":"2026-02-22T23:27:36.327993+0000","last_active":"2026-02-22T23:27:48.190934+0000","last_peered":"2026-02-22T23:27:48.190934+0000","last_clean":"2026-02-22T23:27:48.190934+0000","last_became_active":"2026-02-22T23:27:36.327168+0000","last_became_peered":"2026-02-22T23:27:36.327168+0000","last_unstale":"2026-02-22T23:27:48.190934+0000","last_undegraded":"2026-02-22T23:27:48.190934+0000","last_fullsized":"2026-02-22T23:27:48.190934+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:19:08.163773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8192,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949672967,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":188416,"data_stored":378439,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27128,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344712,"statfs":{"total":96636764160,"available":96608985088,"internally_reserved":0,"allocated":163840,"data_stored":50247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27589064},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27116,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":155648,"data_stored":46460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":192512,"data_stored":386631,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8192,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T23:27:53.306 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:27:53.456 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:53.457 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:27:53.467 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2026-02-22T23:27:52.137589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108544,"kb_used_data":684,"kb_used_omap":6,"kb_used_meta":107769,"kb_avail":377378816,"statfs":{"total":386547056640,"available":386435907584,"internally_reserved":0,"allocated":700416,"data_stored":861777,"data_compressed":3317,"data_compressed_allocated":45056,"data_compressed_original":663552,"omap_allocated":6374,"internal_metadata":110356250},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.956362"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-22T23:27:48.191187+0000","last_change":"2026-02-22T23:27:45.185356+0000","last_active":"2026-02-22T23:27:48.191187+0000","last_peered":"2026-02-22T23:27:48.191187+0000","last_clean":"2026-02-22T23:27:48.191187+0000","last_became_active":"2026-02-22T23:27:45.185140+0000","last_became_peered":"2026-02-22T23:27:45.185140+0000","last_unstale":"2026-02-22T23:27:48.191187+0000","last_undegraded":"2026-02-22T23:27:48.191187+0000","last_fullsized":"2026-02-22T23:27:48.191187+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:44.173266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:44.173266+0000","last_clean_scrub_stamp":"2026-02-22T23:27:44.173266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:59:21.383326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":47,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-22T23:27:48.190934+0000","last_change":"2026-02-22T23:27:36.327993+0000","last_active":"2026-02-22T23:27:48.190934+0000","last_peered":"2026-02-22T23:27:48.190934+0000","last_clean":"2026-02-22T23:27:48.190934+0000","last_became_active":"2026-02-22T23:27:36.327168+0000","last_became_peered":"2026-02-22T23:27:36.327168+0000","last_unstale":"2026-02-22T23:27:48.190934+0000","last_undegraded":"2026-02-22T23:27:48.190934+0000","last_fullsized":"2026-02-22T23:27:48.190934+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T08:19:08.163773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8192,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949672967,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":188416,"data_stored":378439,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27128,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344712,"statfs":{"total":96636764160,"available":96608985088,"internally_reserved":0,"allocated":163840,"data_stored":50247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27589064},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27116,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":155648,"data_stored":46460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":192512,"data_stored":386631,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8192,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T23:27:53.468 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:recovered! 2026-02-22T23:27:53.468 DEBUG:tasks.ec_inconsistent_hinfo:create test object 2026-02-22T23:27:53.468 INFO:tasks.util.rados:rados -p unique_pool_0 put test /etc/group 2026-02-22T23:27:53.468 DEBUG:teuthology.orchestra.run.trial115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put test /etc/group 2026-02-22T23:27:53.495 INFO:tasks.ec_inconsistent_hinfo:remove test object hash info from osd.1 shard and test deep-scrub and repair 2026-02-22T23:27:53.495 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T23:27:53.677 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:53.677 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":19,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:27:48.184904+0000","last_up_change":"2026-02-22T23:27:34.146668+0000","last_in_change":"2026-02-22T23:27:24.830980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T23:27:34.171905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-22T23:27:43.179120+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6809","nonce":3878013816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6810","nonce":3878013816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6812","nonce":3878013816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6811","nonce":3878013816}]},"public_addr":"10.20.193.115:6809/3878013816","cluster_addr":"10.20.193.115:6810/3878013816","heartbeat_back_addr":"10.20.193.115:6812/3878013816","heartbeat_front_addr":"10.20.193.115:6811/3878013816","state":["exists","up"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6805","nonce":464259751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6806","nonce":464259751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6808","nonce":464259751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6807","nonce":464259751}]},"public_addr":"10.20.193.115:6805/464259751","cluster_addr":"10.20.193.115:6806/464259751","heartbeat_back_addr":"10.20.193.115:6808/464259751","heartbeat_front_addr":"10.20.193.115:6807/464259751","state":["exists","up"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6801","nonce":1978117969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6802","nonce":1978117969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6804","nonce":1978117969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6803","nonce":1978117969}]},"public_addr":"10.20.193.115:6801/1978117969","cluster_addr":"10.20.193.115:6802/1978117969","heartbeat_back_addr":"10.20.193.115:6804/1978117969","heartbeat_front_addr":"10.20.193.115:6803/1978117969","state":["exists","up"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6813","nonce":3651482031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6814","nonce":3651482031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6816","nonce":3651482031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6815","nonce":3651482031}]},"public_addr":"10.20.193.115:6813/3651482031","cluster_addr":"10.20.193.115:6814/3651482031","heartbeat_back_addr":"10.20.193.115:6816/3651482031","heartbeat_front_addr":"10.20.193.115:6815/3651482031","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:27:53.688 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json osd map unique_pool_0 test 2026-02-22T23:27:53.844 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:27:53.855 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":19,"pool":"unique_pool_0","pool_id":2,"objname":"test","raw_pgid":"2.40e8aab5","pgid":"2.0","up":[3,1,2],"up_primary":3,"acting":[3,1,2],"acting_primary":3} 2026-02-22T23:27:53.856 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-22T23:27:53.856 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T23:27:56.721 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2026-02-22T23:27:59.957 INFO:tasks.ceph.osd.1:Stopped 2026-02-22T23:27:59.958 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 2.0s1 --op list |grep '"oid":"test"') sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 2.0s1 "$object" rm-attr hinfo_key 2026-02-22T23:27:59.958 DEBUG:teuthology.orchestra.run.trial115:> bash -e -x -c 'object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 2.0s1 --op list |grep '"'"'"oid":"test"'"'"') 2026-02-22T23:27:59.958 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 2.0s1 "$object" rm-attr hinfo_key' 2026-02-22T23:27:59.963 INFO:teuthology.orchestra.run.trial115.stderr:+ '[' -z '' ']' 2026-02-22T23:27:59.963 INFO:teuthology.orchestra.run.trial115.stderr:+ return 2026-02-22T23:27:59.963 INFO:teuthology.orchestra.run.trial115.stderr:+ case $- in 2026-02-22T23:27:59.963 INFO:teuthology.orchestra.run.trial115.stderr:+ return 2026-02-22T23:27:59.964 INFO:teuthology.orchestra.run.trial115.stderr:++ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 2.0s1 --op list 2026-02-22T23:27:59.964 INFO:teuthology.orchestra.run.trial115.stderr:++ grep '"oid":"test"' 2026-02-22T23:28:00.831 INFO:teuthology.orchestra.run.trial115.stderr:+ object='["2.0s1",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":1,"max":0}]' 2026-02-22T23:28:00.831 INFO:teuthology.orchestra.run.trial115.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 2.0s1 '["2.0s1",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":1,"max":0}]' rm-attr hinfo_key 2026-02-22T23:28:02.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2026-02-22T23:28:02.857 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-22T23:28:02.857 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-02-22T23:28:02.904 INFO:tasks.ceph.osd.1:Started 2026-02-22T23:28:02.905 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-22T23:28:02.964 INFO:teuthology.orchestra.run.trial115.stderr:no valid command found; 10 closest matches: 2026-02-22T23:28:02.965 INFO:teuthology.orchestra.run.trial115.stderr:0 2026-02-22T23:28:02.965 INFO:teuthology.orchestra.run.trial115.stderr:1 2026-02-22T23:28:02.965 INFO:teuthology.orchestra.run.trial115.stderr:2 2026-02-22T23:28:02.965 INFO:teuthology.orchestra.run.trial115.stderr:abort 2026-02-22T23:28:02.965 INFO:teuthology.orchestra.run.trial115.stderr:assert 2026-02-22T23:28:02.965 INFO:teuthology.orchestra.run.trial115.stderr:config diff 2026-02-22T23:28:02.965 INFO:teuthology.orchestra.run.trial115.stderr:config diff get 2026-02-22T23:28:02.965 INFO:teuthology.orchestra.run.trial115.stderr:config get 2026-02-22T23:28:02.965 INFO:teuthology.orchestra.run.trial115.stderr:config help [] 2026-02-22T23:28:02.966 INFO:teuthology.orchestra.run.trial115.stderr:config set ... 2026-02-22T23:28:02.966 INFO:teuthology.orchestra.run.trial115.stderr:admin_socket: invalid command 2026-02-22T23:28:02.967 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-22T23:28:02.967 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-22T23:28:04.236 INFO:tasks.ceph.osd.1.trial115.stderr:2026-02-22T23:28:04.233+0000 7f33e486b8c0 -1 Falling back to public interface 2026-02-22T23:28:07.968 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-22T23:28:08.028 INFO:teuthology.orchestra.run.trial115.stderr:no valid command found; 10 closest matches: 2026-02-22T23:28:08.028 INFO:teuthology.orchestra.run.trial115.stderr:0 2026-02-22T23:28:08.028 INFO:teuthology.orchestra.run.trial115.stderr:1 2026-02-22T23:28:08.029 INFO:teuthology.orchestra.run.trial115.stderr:2 2026-02-22T23:28:08.029 INFO:teuthology.orchestra.run.trial115.stderr:abort 2026-02-22T23:28:08.029 INFO:teuthology.orchestra.run.trial115.stderr:assert 2026-02-22T23:28:08.029 INFO:teuthology.orchestra.run.trial115.stderr:bluestore allocator dump block 2026-02-22T23:28:08.029 INFO:teuthology.orchestra.run.trial115.stderr:bluestore allocator fragmentation block 2026-02-22T23:28:08.029 INFO:teuthology.orchestra.run.trial115.stderr:bluestore allocator fragmentation histogram block [] [] 2026-02-22T23:28:08.029 INFO:teuthology.orchestra.run.trial115.stderr:bluestore allocator score block 2026-02-22T23:28:08.029 INFO:teuthology.orchestra.run.trial115.stderr:config diff 2026-02-22T23:28:08.029 INFO:teuthology.orchestra.run.trial115.stderr:admin_socket: invalid command 2026-02-22T23:28:08.030 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-22T23:28:08.031 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-22T23:28:08.099 INFO:tasks.ceph.osd.1.trial115.stderr:2026-02-22T23:28:08.097+0000 7f33e486b8c0 -1 osd.1 19 log_to_monitors true 2026-02-22T23:28:08.157 INFO:tasks.ceph.osd.1.trial115.stderr:2026-02-22T23:28:08.157+0000 7f33dc64f640 -1 osd.1 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T23:28:13.032 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-22T23:28:13.097 INFO:teuthology.orchestra.run.trial115.stdout:{ 2026-02-22T23:28:13.097 INFO:teuthology.orchestra.run.trial115.stdout: "ops": [], 2026-02-22T23:28:13.098 INFO:teuthology.orchestra.run.trial115.stdout: "num_ops": 0 2026-02-22T23:28:13.098 INFO:teuthology.orchestra.run.trial115.stdout:} 2026-02-22T23:28:13.104 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for osd.1 to be up 2026-02-22T23:28:13.105 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T23:28:13.253 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:28:13.253 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":24,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:28:10.161170+0000","last_up_change":"2026-02-22T23:28:09.157993+0000","last_in_change":"2026-02-22T23:27:24.830980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T23:27:34.171905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-22T23:27:43.179120+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6809","nonce":3878013816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6810","nonce":3878013816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6812","nonce":3878013816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6811","nonce":3878013816}]},"public_addr":"10.20.193.115:6809/3878013816","cluster_addr":"10.20.193.115:6810/3878013816","heartbeat_back_addr":"10.20.193.115:6812/3878013816","heartbeat_front_addr":"10.20.193.115:6811/3878013816","state":["exists","up"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":23,"up_thru":23,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6805","nonce":2409177103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6806","nonce":2409177103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6808","nonce":2409177103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6807","nonce":2409177103}]},"public_addr":"10.20.193.115:6805/2409177103","cluster_addr":"10.20.193.115:6806/2409177103","heartbeat_back_addr":"10.20.193.115:6808/2409177103","heartbeat_front_addr":"10.20.193.115:6807/2409177103","state":["exists","up"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6801","nonce":1978117969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6802","nonce":1978117969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6804","nonce":1978117969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6803","nonce":1978117969}]},"public_addr":"10.20.193.115:6801/1978117969","cluster_addr":"10.20.193.115:6802/1978117969","heartbeat_back_addr":"10.20.193.115:6804/1978117969","heartbeat_front_addr":"10.20.193.115:6803/1978117969","state":["exists","up"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6813","nonce":3651482031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6814","nonce":3651482031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6816","nonce":3651482031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6815","nonce":3651482031}]},"public_addr":"10.20.193.115:6813/3651482031","cluster_addr":"10.20.193.115:6814/3651482031","heartbeat_back_addr":"10.20.193.115:6816/3651482031","heartbeat_front_addr":"10.20.193.115:6815/3651482031","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:28:13.266 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:osd.1 is up 2026-02-22T23:28:13.266 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2026-02-22T23:28:13.409 INFO:teuthology.orchestra.run.trial115.stderr:instructing pg 2.0s0 on osd.3 to deep-scrub 2026-02-22T23:28:13.420 DEBUG:tasks.ec_inconsistent_hinfo:waiting for pg 2.0 deep-scrub complete (check_time_now=1771802893) 2026-02-22T23:28:14.296 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:28:14.297+0000 7f4d23f14640 -1 log_channel(cluster) log [ERR] : 2.0 shard 1(1) soid 2:ad551702:::test:head : candidate had a read error, candidate had a missing hinfo key 2026-02-22T23:28:14.296 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:28:14.297+0000 7f4d23f14640 -1 log_channel(cluster) log [ERR] : 2.0s0 deep-scrub 0 missing, 1 inconsistent objects 2026-02-22T23:28:14.296 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:28:14.297+0000 7f4d23f14640 -1 log_channel(cluster) log [ERR] : 2.0 deep-scrub 1 errors 2026-02-22T23:28:18.422 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:28:18.423 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:28:18.423 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:28:18.424 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:28:18.510 INFO:teuthology.orchestra.run.trial115.stdout:98784247812 2026-02-22T23:28:18.510 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:28:18.512 INFO:teuthology.orchestra.run.trial115.stdout:38654705676 2026-02-22T23:28:18.513 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:28:18.514 INFO:teuthology.orchestra.run.trial115.stdout:38654705676 2026-02-22T23:28:18.515 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:28:18.518 INFO:teuthology.orchestra.run.trial115.stdout:42949672973 2026-02-22T23:28:18.519 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:28:18.665 INFO:teuthology.orchestra.run.trial115.stdout:98784247810 2026-02-22T23:28:18.678 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247812 got 98784247810 for osd.1 2026-02-22T23:28:18.701 INFO:teuthology.orchestra.run.trial115.stdout:38654705675 2026-02-22T23:28:18.702 INFO:teuthology.orchestra.run.trial115.stdout:38654705675 2026-02-22T23:28:18.711 INFO:teuthology.orchestra.run.trial115.stdout:42949672972 2026-02-22T23:28:18.713 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705676 got 38654705675 for osd.2 2026-02-22T23:28:18.714 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705676 got 38654705675 for osd.3 2026-02-22T23:28:18.724 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672973 got 42949672972 for osd.0 2026-02-22T23:28:19.679 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:28:19.714 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:28:19.714 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:28:19.724 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:28:19.837 INFO:teuthology.orchestra.run.trial115.stdout:98784247810 2026-02-22T23:28:19.849 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247812 got 98784247810 for osd.1 2026-02-22T23:28:19.866 INFO:teuthology.orchestra.run.trial115.stdout:38654705675 2026-02-22T23:28:19.867 INFO:teuthology.orchestra.run.trial115.stdout:38654705675 2026-02-22T23:28:19.879 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705676 got 38654705675 for osd.3 2026-02-22T23:28:19.880 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705676 got 38654705675 for osd.2 2026-02-22T23:28:19.913 INFO:teuthology.orchestra.run.trial115.stdout:42949672972 2026-02-22T23:28:19.925 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672973 got 42949672972 for osd.0 2026-02-22T23:28:20.225 INFO:tasks.ceph.mon.a.trial115.stderr:2026-02-22T23:28:20.225+0000 7f2aed71d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 scrub errors (OSD_SCRUB_ERRORS) 2026-02-22T23:28:20.225 INFO:tasks.ceph.mon.a.trial115.stderr:2026-02-22T23:28:20.225+0000 7f2aed71d640 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 1 pg inconsistent (PG_DAMAGED) 2026-02-22T23:28:20.850 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:28:20.880 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:28:20.881 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:28:20.926 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:28:21.016 INFO:teuthology.orchestra.run.trial115.stdout:98784247812 2026-02-22T23:28:21.030 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247812 got 98784247812 for osd.1 2026-02-22T23:28:21.030 DEBUG:teuthology.parallel:result is None 2026-02-22T23:28:21.039 INFO:teuthology.orchestra.run.trial115.stdout:38654705677 2026-02-22T23:28:21.052 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705676 got 38654705677 for osd.3 2026-02-22T23:28:21.053 DEBUG:teuthology.parallel:result is None 2026-02-22T23:28:21.065 INFO:teuthology.orchestra.run.trial115.stdout:38654705677 2026-02-22T23:28:21.079 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705676 got 38654705677 for osd.2 2026-02-22T23:28:21.079 DEBUG:teuthology.parallel:result is None 2026-02-22T23:28:21.088 INFO:teuthology.orchestra.run.trial115.stdout:42949672973 2026-02-22T23:28:21.100 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672973 got 42949672973 for osd.0 2026-02-22T23:28:21.101 DEBUG:teuthology.parallel:result is None 2026-02-22T23:28:21.101 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:28:21.287 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:28:21.288 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:28:21.298 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2026-02-22T23:28:20.140521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":344950,"num_objects":4,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":39,"num_write_kb":340,"num_scrub_errors":1,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":1,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":109316,"kb_used_data":872,"kb_used_omap":8,"kb_used_meta":108343,"kb_avail":377378044,"statfs":{"total":386547056640,"available":386435117056,"internally_reserved":0,"allocated":892928,"data_stored":979908,"data_compressed":17294,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":9177,"internal_metadata":110943271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":1,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":1,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001144"},"pg_stats":[{"pgid":"2.0","version":"19'2","reported_seq":46,"reported_epoch":24,"state":"active+clean+inconsistent","last_fresh":"2026-02-22T23:28:14.298439+0000","last_change":"2026-02-22T23:28:14.298461+0000","last_active":"2026-02-22T23:28:14.298439+0000","last_peered":"2026-02-22T23:28:14.298439+0000","last_clean":"2026-02-22T23:28:14.298439+0000","last_became_active":"2026-02-22T23:28:10.168198+0000","last_became_peered":"2026-02-22T23:28:10.168198+0000","last_unstale":"2026-02-22T23:28:14.298439+0000","last_undegraded":"2026-02-22T23:28:14.298439+0000","last_fullsized":"2026-02-22T23:28:14.298439+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:14.298392+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:14.298392+0000","last_clean_scrub_stamp":"2026-02-22T23:27:44.173266+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:09:07.391992+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16758,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":18,"num_scrub_errors":1,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":1,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-22T23:28:10.166195+0000","last_change":"2026-02-22T23:28:10.166195+0000","last_active":"2026-02-22T23:28:10.166195+0000","last_peered":"2026-02-22T23:28:10.166195+0000","last_clean":"2026-02-22T23:28:10.166195+0000","last_became_active":"2026-02-22T23:28:10.166025+0000","last_became_peered":"2026-02-22T23:28:10.166025+0000","last_unstale":"2026-02-22T23:28:10.166195+0000","last_undegraded":"2026-02-22T23:28:10.166195+0000","last_fullsized":"2026-02-22T23:28:10.166195+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:41:03.416177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":16758,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":18,"num_scrub_errors":1,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":1,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":49152,"data_compressed":16108,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949672973,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":229376,"data_stored":396785,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705677,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":212992,"data_stored":84977,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":27589064},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705677,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":212992,"data_stored":84977,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247812,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":232,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":237568,"data_stored":413169,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":28176083},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16384,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16384,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16384,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T23:28:21.299 DEBUG:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "19'2", 'reported_seq': 46, 'reported_epoch': 24, 'state': 'active+clean+inconsistent', 'last_fresh': '2026-02-22T23:28:14.298439+0000', 'last_change': '2026-02-22T23:28:14.298461+0000', 'last_active': '2026-02-22T23:28:14.298439+0000', 'last_peered': '2026-02-22T23:28:14.298439+0000', 'last_clean': '2026-02-22T23:28:14.298439+0000', 'last_became_active': '2026-02-22T23:28:10.168198+0000', 'last_became_peered': '2026-02-22T23:28:10.168198+0000', 'last_unstale': '2026-02-22T23:28:14.298439+0000', 'last_undegraded': '2026-02-22T23:28:14.298439+0000', 'last_fullsized': '2026-02-22T23:28:14.298439+0000', 'mapping_epoch': 23, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 24, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:14.298392+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:14.298392+0000', 'last_clean_scrub_stamp': '2026-02-22T23:27:44.173266+0000', 'objects_scrubbed': 2, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-24T03:09:07.391992+0000', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16758, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 18, 'num_scrub_errors': 1, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 1, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:28:21.300 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg repair 2.0 2026-02-22T23:28:21.449 INFO:teuthology.orchestra.run.trial115.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-22T23:28:21.460 DEBUG:tasks.ec_inconsistent_hinfo:waiting for pg 2.0 deep-scrub complete (check_time_now=1771802901) 2026-02-22T23:28:22.256 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:28:22.257+0000 7f4d23f14640 -1 log_channel(cluster) log [ERR] : 2.0 shard 1(1) soid 2:ad551702:::test:head : candidate had a read error, candidate had a missing hinfo key 2026-02-22T23:28:22.256 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:28:22.257+0000 7f4d23f14640 -1 log_channel(cluster) log [ERR] : 2.0s0 repair 0 missing, 1 inconsistent objects 2026-02-22T23:28:22.256 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:28:22.257+0000 7f4d23f14640 -1 log_channel(cluster) log [ERR] : 2.0s0 repair 0 missing, 1 inconsistent objects 2026-02-22T23:28:22.256 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:28:22.257+0000 7f4d23f14640 -1 log_channel(cluster) log [ERR] : 2.0 repair 1 errors, 1 fixed 2026-02-22T23:28:26.461 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:28:26.462 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:28:26.462 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:28:26.463 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:28:26.550 INFO:teuthology.orchestra.run.trial115.stdout:38654705679 2026-02-22T23:28:26.551 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:28:26.552 INFO:teuthology.orchestra.run.trial115.stdout:98784247814 2026-02-22T23:28:26.552 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:28:26.553 INFO:teuthology.orchestra.run.trial115.stdout:38654705679 2026-02-22T23:28:26.554 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:28:26.558 INFO:teuthology.orchestra.run.trial115.stdout:42949672976 2026-02-22T23:28:26.559 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:28:26.702 INFO:teuthology.orchestra.run.trial115.stdout:98784247813 2026-02-22T23:28:26.705 INFO:teuthology.orchestra.run.trial115.stdout:38654705678 2026-02-22T23:28:26.714 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247814 got 98784247813 for osd.1 2026-02-22T23:28:26.718 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705679 got 38654705678 for osd.3 2026-02-22T23:28:26.746 INFO:teuthology.orchestra.run.trial115.stdout:38654705678 2026-02-22T23:28:26.751 INFO:teuthology.orchestra.run.trial115.stdout:42949672975 2026-02-22T23:28:26.759 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705679 got 38654705678 for osd.2 2026-02-22T23:28:26.765 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672976 got 42949672975 for osd.0 2026-02-22T23:28:27.716 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:28:27.718 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:28:27.759 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:28:27.766 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:28:27.880 INFO:teuthology.orchestra.run.trial115.stdout:98784247813 2026-02-22T23:28:27.892 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247814 got 98784247813 for osd.1 2026-02-22T23:28:27.906 INFO:teuthology.orchestra.run.trial115.stdout:38654705678 2026-02-22T23:28:27.914 INFO:teuthology.orchestra.run.trial115.stdout:38654705678 2026-02-22T23:28:27.918 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705679 got 38654705678 for osd.2 2026-02-22T23:28:27.926 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705679 got 38654705678 for osd.3 2026-02-22T23:28:27.945 INFO:teuthology.orchestra.run.trial115.stdout:42949672975 2026-02-22T23:28:27.957 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672976 got 42949672975 for osd.0 2026-02-22T23:28:28.893 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:28:28.919 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:28:28.927 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:28:28.957 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:28:29.048 INFO:teuthology.orchestra.run.trial115.stdout:98784247814 2026-02-22T23:28:29.061 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247814 got 98784247814 for osd.1 2026-02-22T23:28:29.062 DEBUG:teuthology.parallel:result is None 2026-02-22T23:28:29.085 INFO:teuthology.orchestra.run.trial115.stdout:38654705679 2026-02-22T23:28:29.092 INFO:teuthology.orchestra.run.trial115.stdout:38654705679 2026-02-22T23:28:29.098 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705679 got 38654705679 for osd.3 2026-02-22T23:28:29.098 DEBUG:teuthology.parallel:result is None 2026-02-22T23:28:29.105 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705679 got 38654705679 for osd.2 2026-02-22T23:28:29.105 DEBUG:teuthology.parallel:result is None 2026-02-22T23:28:29.134 INFO:teuthology.orchestra.run.trial115.stdout:42949672976 2026-02-22T23:28:29.146 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672976 got 42949672976 for osd.0 2026-02-22T23:28:29.146 DEBUG:teuthology.parallel:result is None 2026-02-22T23:28:29.146 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:28:29.327 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:28:29.328 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:28:29.339 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":48,"stamp":"2026-02-22T23:28:28.141472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":344950,"num_objects":4,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":39,"num_write_kb":340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8379,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":109300,"kb_used_data":872,"kb_used_omap":8,"kb_used_meta":108343,"kb_avail":377378060,"statfs":{"total":386547056640,"available":386435133440,"internally_reserved":0,"allocated":892928,"data_stored":979908,"data_compressed":17294,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":9177,"internal_metadata":110943271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8379,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001342"},"pg_stats":[{"pgid":"2.0","version":"19'2","reported_seq":62,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-22T23:28:22.262661+0000","last_change":"2026-02-22T23:28:22.263007+0000","last_active":"2026-02-22T23:28:22.262661+0000","last_peered":"2026-02-22T23:28:22.262661+0000","last_clean":"2026-02-22T23:28:22.262661+0000","last_became_active":"2026-02-22T23:28:10.168198+0000","last_became_peered":"2026-02-22T23:28:10.168198+0000","last_unstale":"2026-02-22T23:28:22.262661+0000","last_undegraded":"2026-02-22T23:28:22.262661+0000","last_fullsized":"2026-02-22T23:28:22.262661+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T04:13:35.715084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16758,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8379,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-22T23:28:10.166195+0000","last_change":"2026-02-22T23:28:10.166195+0000","last_active":"2026-02-22T23:28:10.166195+0000","last_peered":"2026-02-22T23:28:10.166195+0000","last_clean":"2026-02-22T23:28:10.166195+0000","last_became_active":"2026-02-22T23:28:10.166025+0000","last_became_peered":"2026-02-22T23:28:10.166025+0000","last_unstale":"2026-02-22T23:28:10.166195+0000","last_undegraded":"2026-02-22T23:28:10.166195+0000","last_fullsized":"2026-02-22T23:28:10.166195+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:41:03.416177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":16758,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8379,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":49152,"data_compressed":16108,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949672976,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":229376,"data_stored":396785,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705679,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":212992,"data_stored":84977,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":27589064},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705679,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":212992,"data_stored":84977,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247814,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":232,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":237568,"data_stored":413169,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":28176083},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16384,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16384,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16384,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-22T23:28:29.340 DEBUG:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "19'2", 'reported_seq': 62, 'reported_epoch': 24, 'state': 'active+clean', 'last_fresh': '2026-02-22T23:28:22.262661+0000', 'last_change': '2026-02-22T23:28:22.263007+0000', 'last_active': '2026-02-22T23:28:22.262661+0000', 'last_peered': '2026-02-22T23:28:22.262661+0000', 'last_clean': '2026-02-22T23:28:22.262661+0000', 'last_became_active': '2026-02-22T23:28:10.168198+0000', 'last_became_peered': '2026-02-22T23:28:10.168198+0000', 'last_unstale': '2026-02-22T23:28:22.262661+0000', 'last_undegraded': '2026-02-22T23:28:22.262661+0000', 'last_fullsized': '2026-02-22T23:28:22.262661+0000', 'mapping_epoch': 23, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 24, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-24T04:13:35.715084+0000', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16758, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 18, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 8379, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 2], 'acting': [3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:28:29.341 INFO:tasks.ec_inconsistent_hinfo:remove test object hash info from primary osd.3 shard and test backfill 2026-02-22T23:28:29.341 DEBUG:tasks.ec_inconsistent_hinfo:write some data 2026-02-22T23:28:29.341 INFO:tasks.util.rados:rados -p unique_pool_0 bench 30 write -b 4096 --no-cleanup 2026-02-22T23:28:29.342 DEBUG:teuthology.orchestra.run.trial115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 bench 30 write -b 4096 --no-cleanup 2026-02-22T23:28:29.365 INFO:teuthology.orchestra.run.trial115.stderr:INFO: op_size has been rounded to 8192 2026-02-22T23:28:29.366 INFO:teuthology.orchestra.run.trial115.stdout:hints = 1 2026-02-22T23:28:29.366 INFO:teuthology.orchestra.run.trial115.stdout:Maintaining 16 concurrent writes of 8192 bytes to objects of size 8192 for up to 30 seconds or 0 objects 2026-02-22T23:28:29.366 INFO:teuthology.orchestra.run.trial115.stdout:Object prefix: benchmark_data_trial115_25337 2026-02-22T23:28:29.366 INFO:teuthology.orchestra.run.trial115.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-22T23:28:29.366 INFO:teuthology.orchestra.run.trial115.stdout: 0 0 0 0 0 0 - 0 2026-02-22T23:28:30.365 INFO:teuthology.orchestra.run.trial115.stdout: 1 16 2057 2041 15.9442 15.9453 0.00508741 0.00781136 2026-02-22T23:28:31.365 INFO:teuthology.orchestra.run.trial115.stdout: 2 16 4517 4501 17.5809 19.2188 0.00667186 0.00709773 2026-02-22T23:28:32.365 INFO:teuthology.orchestra.run.trial115.stdout: 3 16 6891 6875 17.9024 18.5469 0.00501378 0.00697017 2026-02-22T23:28:33.365 INFO:teuthology.orchestra.run.trial115.stdout: 4 16 9383 9367 18.2937 19.4688 0.00671254 0.00682601 2026-02-22T23:28:34.366 INFO:teuthology.orchestra.run.trial115.stdout: 5 16 11827 11811 18.4534 19.0938 0.00707623 0.00676653 2026-02-22T23:28:35.366 INFO:teuthology.orchestra.run.trial115.stdout: 6 16 14219 14203 18.4922 18.6875 0.00584296 0.00675442 2026-02-22T23:28:36.366 INFO:teuthology.orchestra.run.trial115.stdout: 7 16 16764 16748 18.6906 19.8828 0.00590844 0.00668319 2026-02-22T23:28:37.366 INFO:teuthology.orchestra.run.trial115.stdout: 8 16 19154 19138 18.6881 18.6719 0.00659358 0.00668288 2026-02-22T23:28:38.366 INFO:teuthology.orchestra.run.trial115.stdout: 9 16 21567 21551 18.7061 18.8516 0.00568879 0.00667907 2026-02-22T23:28:39.366 INFO:teuthology.orchestra.run.trial115.stdout: 10 16 24071 24055 18.7916 19.5625 0.00560004 0.00664833 2026-02-22T23:28:40.366 INFO:teuthology.orchestra.run.trial115.stdout: 11 16 26383 26367 18.7252 18.0625 0.00524413 0.00667257 2026-02-22T23:28:41.366 INFO:teuthology.orchestra.run.trial115.stdout: 12 16 28631 28615 18.6282 17.5625 0.00583352 0.00670706 2026-02-22T23:28:42.366 INFO:teuthology.orchestra.run.trial115.stdout: 13 16 30872 30856 18.5419 17.5078 0.00649561 0.00673805 2026-02-22T23:28:43.366 INFO:teuthology.orchestra.run.trial115.stdout: 14 16 33072 33056 18.4451 17.1875 0.00531288 0.00677402 2026-02-22T23:28:44.366 INFO:teuthology.orchestra.run.trial115.stdout: 15 16 35167 35151 18.3065 16.3672 0.00728071 0.00682498 2026-02-22T23:28:45.366 INFO:teuthology.orchestra.run.trial115.stdout: 16 16 37389 37373 18.2472 17.3594 0.00675988 0.00684728 2026-02-22T23:28:46.366 INFO:teuthology.orchestra.run.trial115.stdout: 17 16 39581 39565 18.1812 17.125 0.00739112 0.00687217 2026-02-22T23:28:47.367 INFO:teuthology.orchestra.run.trial115.stdout: 18 16 41957 41941 18.2023 18.5625 0.00723966 0.00686401 2026-02-22T23:28:48.367 INFO:teuthology.orchestra.run.trial115.stdout: 19 16 44193 44177 18.1636 17.4688 0.00659815 0.00687909 2026-02-22T23:28:49.367 INFO:teuthology.orchestra.run.trial115.stdout:2026-02-22T23:28:49.369279+0000 min lat: 0.00145796 max lat: 0.0159508 avg lat: 0.00687656 2026-02-22T23:28:49.367 INFO:teuthology.orchestra.run.trial115.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-22T23:28:49.367 INFO:teuthology.orchestra.run.trial115.stdout: 20 16 46536 46520 18.1706 18.3047 0.00775477 0.00687656 2026-02-22T23:28:50.367 INFO:teuthology.orchestra.run.trial115.stdout: 21 16 48708 48692 18.1133 16.9688 0.00630848 0.00689834 2026-02-22T23:28:51.367 INFO:teuthology.orchestra.run.trial115.stdout: 22 16 50936 50920 18.0811 17.4062 0.00726379 0.00691056 2026-02-22T23:28:52.367 INFO:teuthology.orchestra.run.trial115.stdout: 23 16 53099 53083 18.0296 16.8984 0.00740327 0.00693033 2026-02-22T23:28:53.367 INFO:teuthology.orchestra.run.trial115.stdout: 24 16 55312 55296 17.9987 17.2891 0.00626647 0.00694233 2026-02-22T23:28:54.367 INFO:teuthology.orchestra.run.trial115.stdout: 25 16 57568 57552 17.9837 17.625 0.00908868 0.00694759 2026-02-22T23:28:55.367 INFO:teuthology.orchestra.run.trial115.stdout: 26 16 59885 59869 17.9882 18.1016 0.00565409 0.00694643 2026-02-22T23:28:56.367 INFO:teuthology.orchestra.run.trial115.stdout: 27 16 62130 62114 17.9715 17.5391 0.00774646 0.00695285 2026-02-22T23:28:57.367 INFO:teuthology.orchestra.run.trial115.stdout: 28 16 64412 64396 17.9663 17.8281 0.00401649 0.00695501 2026-02-22T23:28:58.367 INFO:teuthology.orchestra.run.trial115.stdout: 29 16 66529 66513 17.9171 16.5391 0.00614385 0.00697425 2026-02-22T23:28:59.367 INFO:teuthology.orchestra.run.trial115.stdout: 30 11 68596 68585 17.8594 16.1875 0.00774203 0.00699686 2026-02-22T23:29:00.368 INFO:teuthology.orchestra.run.trial115.stdout:Total time run: 30.0063 2026-02-22T23:29:00.368 INFO:teuthology.orchestra.run.trial115.stdout:Total writes made: 68596 2026-02-22T23:29:00.368 INFO:teuthology.orchestra.run.trial115.stdout:Write size: 8192 2026-02-22T23:29:00.368 INFO:teuthology.orchestra.run.trial115.stdout:Object size: 8192 2026-02-22T23:29:00.368 INFO:teuthology.orchestra.run.trial115.stdout:Bandwidth (MB/sec): 17.8598 2026-02-22T23:29:00.368 INFO:teuthology.orchestra.run.trial115.stdout:Stddev Bandwidth: 1.03377 2026-02-22T23:29:00.368 INFO:teuthology.orchestra.run.trial115.stdout:Max bandwidth (MB/sec): 19.8828 2026-02-22T23:29:00.368 INFO:teuthology.orchestra.run.trial115.stdout:Min bandwidth (MB/sec): 15.9453 2026-02-22T23:29:00.369 INFO:teuthology.orchestra.run.trial115.stdout:Average IOPS: 2286 2026-02-22T23:29:00.369 INFO:teuthology.orchestra.run.trial115.stdout:Stddev IOPS: 132.322 2026-02-22T23:29:00.369 INFO:teuthology.orchestra.run.trial115.stdout:Max IOPS: 2545 2026-02-22T23:29:00.369 INFO:teuthology.orchestra.run.trial115.stdout:Min IOPS: 2041 2026-02-22T23:29:00.369 INFO:teuthology.orchestra.run.trial115.stdout:Average Latency(s): 0.00699687 2026-02-22T23:29:00.369 INFO:teuthology.orchestra.run.trial115.stdout:Stddev Latency(s): 0.0013179 2026-02-22T23:29:00.369 INFO:teuthology.orchestra.run.trial115.stdout:Max latency(s): 0.0159508 2026-02-22T23:29:00.369 INFO:teuthology.orchestra.run.trial115.stdout:Min latency(s): 0.00145796 2026-02-22T23:29:00.373 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T23:29:00.559 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:29:00.559 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":25,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:28:41.228337+0000","last_up_change":"2026-02-22T23:28:09.157993+0000","last_in_change":"2026-02-22T23:27:24.830980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T23:27:34.171905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-22T23:27:43.179120+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6809","nonce":3878013816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6810","nonce":3878013816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6812","nonce":3878013816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6811","nonce":3878013816}]},"public_addr":"10.20.193.115:6809/3878013816","cluster_addr":"10.20.193.115:6810/3878013816","heartbeat_back_addr":"10.20.193.115:6812/3878013816","heartbeat_front_addr":"10.20.193.115:6811/3878013816","state":["exists","up"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":23,"up_thru":23,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6805","nonce":2409177103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6806","nonce":2409177103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6808","nonce":2409177103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6807","nonce":2409177103}]},"public_addr":"10.20.193.115:6805/2409177103","cluster_addr":"10.20.193.115:6806/2409177103","heartbeat_back_addr":"10.20.193.115:6808/2409177103","heartbeat_front_addr":"10.20.193.115:6807/2409177103","state":["exists","up"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6801","nonce":1978117969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6802","nonce":1978117969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6804","nonce":1978117969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6803","nonce":1978117969}]},"public_addr":"10.20.193.115:6801/1978117969","cluster_addr":"10.20.193.115:6802/1978117969","heartbeat_back_addr":"10.20.193.115:6804/1978117969","heartbeat_front_addr":"10.20.193.115:6803/1978117969","state":["exists","up"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6813","nonce":3651482031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6814","nonce":3651482031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6816","nonce":3651482031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6815","nonce":3651482031}]},"public_addr":"10.20.193.115:6813/3651482031","cluster_addr":"10.20.193.115:6814/3651482031","heartbeat_back_addr":"10.20.193.115:6816/3651482031","heartbeat_front_addr":"10.20.193.115:6815/3651482031","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:32.314887+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:31.613759+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:31.523737+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:32.017476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:29:00.570 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json osd map unique_pool_0 test 2026-02-22T23:29:00.715 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:29:00.725 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":25,"pool":"unique_pool_0","pool_id":2,"objname":"test","raw_pgid":"2.40e8aab5","pgid":"2.0","up":[3,1,2],"up_primary":3,"acting":[3,1,2],"acting_primary":3} 2026-02-22T23:29:00.725 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-02-22T23:29:00.726 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T23:29:06.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2026-02-22T23:29:06.827 INFO:tasks.ceph.osd.3:Stopped 2026-02-22T23:29:06.828 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 --op list |grep '"oid":"test"') sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 "$object" rm-attr hinfo_key 2026-02-22T23:29:06.828 DEBUG:teuthology.orchestra.run.trial115:> bash -e -x -c 'object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 --op list |grep '"'"'"oid":"test"'"'"') 2026-02-22T23:29:06.828 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 "$object" rm-attr hinfo_key' 2026-02-22T23:29:06.833 INFO:teuthology.orchestra.run.trial115.stderr:+ '[' -z '' ']' 2026-02-22T23:29:06.834 INFO:teuthology.orchestra.run.trial115.stderr:+ return 2026-02-22T23:29:06.834 INFO:teuthology.orchestra.run.trial115.stderr:+ case $- in 2026-02-22T23:29:06.834 INFO:teuthology.orchestra.run.trial115.stderr:+ return 2026-02-22T23:29:06.835 INFO:teuthology.orchestra.run.trial115.stderr:++ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 --op list 2026-02-22T23:29:06.835 INFO:teuthology.orchestra.run.trial115.stderr:++ grep '"oid":"test"' 2026-02-22T23:29:07.857 INFO:teuthology.orchestra.run.trial115.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-22T23:29:09.417 INFO:teuthology.orchestra.run.trial115.stderr:+ object='["2.0s0",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":0,"max":0}]' 2026-02-22T23:29:09.417 INFO:teuthology.orchestra.run.trial115.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 '["2.0s0",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":0,"max":0}]' rm-attr hinfo_key 2026-02-22T23:29:11.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2026-02-22T23:29:14.833 INFO:tasks.ceph.osd.3:Restarting daemon 2026-02-22T23:29:14.833 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2026-02-22T23:29:14.876 INFO:tasks.ceph.osd.3:Started 2026-02-22T23:29:14.877 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2026-02-22T23:29:14.935 INFO:teuthology.orchestra.run.trial115.stderr:no valid command found; 10 closest matches: 2026-02-22T23:29:14.935 INFO:teuthology.orchestra.run.trial115.stderr:0 2026-02-22T23:29:14.935 INFO:teuthology.orchestra.run.trial115.stderr:1 2026-02-22T23:29:14.935 INFO:teuthology.orchestra.run.trial115.stderr:2 2026-02-22T23:29:14.935 INFO:teuthology.orchestra.run.trial115.stderr:abort 2026-02-22T23:29:14.935 INFO:teuthology.orchestra.run.trial115.stderr:assert 2026-02-22T23:29:14.936 INFO:teuthology.orchestra.run.trial115.stderr:config diff 2026-02-22T23:29:14.936 INFO:teuthology.orchestra.run.trial115.stderr:config diff get 2026-02-22T23:29:14.936 INFO:teuthology.orchestra.run.trial115.stderr:config get 2026-02-22T23:29:14.936 INFO:teuthology.orchestra.run.trial115.stderr:config help [] 2026-02-22T23:29:14.936 INFO:teuthology.orchestra.run.trial115.stderr:config set ... 2026-02-22T23:29:14.936 INFO:teuthology.orchestra.run.trial115.stderr:admin_socket: invalid command 2026-02-22T23:29:14.937 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-22T23:29:14.938 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2026-02-22T23:29:16.188 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:29:16.188+0000 7fd82b5b88c0 -1 Falling back to public interface 2026-02-22T23:29:19.938 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2026-02-22T23:29:20.000 INFO:teuthology.orchestra.run.trial115.stderr:no valid command found; 10 closest matches: 2026-02-22T23:29:20.000 INFO:teuthology.orchestra.run.trial115.stderr:0 2026-02-22T23:29:20.000 INFO:teuthology.orchestra.run.trial115.stderr:1 2026-02-22T23:29:20.001 INFO:teuthology.orchestra.run.trial115.stderr:2 2026-02-22T23:29:20.001 INFO:teuthology.orchestra.run.trial115.stderr:abort 2026-02-22T23:29:20.001 INFO:teuthology.orchestra.run.trial115.stderr:assert 2026-02-22T23:29:20.001 INFO:teuthology.orchestra.run.trial115.stderr:bluestore allocator dump block 2026-02-22T23:29:20.001 INFO:teuthology.orchestra.run.trial115.stderr:bluestore allocator fragmentation block 2026-02-22T23:29:20.001 INFO:teuthology.orchestra.run.trial115.stderr:bluestore allocator fragmentation histogram block [] [] 2026-02-22T23:29:20.001 INFO:teuthology.orchestra.run.trial115.stderr:bluestore allocator score block 2026-02-22T23:29:20.001 INFO:teuthology.orchestra.run.trial115.stderr:config diff 2026-02-22T23:29:20.001 INFO:teuthology.orchestra.run.trial115.stderr:admin_socket: invalid command 2026-02-22T23:29:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-22T23:29:20.003 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2026-02-22T23:29:20.303 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:29:20.300+0000 7fd82b5b88c0 -1 osd.3 25 log_to_monitors true 2026-02-22T23:29:21.236 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:29:21.236+0000 7fd81f394640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T23:29:25.003 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2026-02-22T23:29:25.069 INFO:teuthology.orchestra.run.trial115.stdout:{ 2026-02-22T23:29:25.069 INFO:teuthology.orchestra.run.trial115.stdout: "ops": [], 2026-02-22T23:29:25.070 INFO:teuthology.orchestra.run.trial115.stdout: "num_ops": 0 2026-02-22T23:29:25.070 INFO:teuthology.orchestra.run.trial115.stdout:} 2026-02-22T23:29:25.077 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for osd.3 to be up 2026-02-22T23:29:25.077 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T23:29:25.226 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:29:25.226 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":30,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:29:23.237972+0000","last_up_change":"2026-02-22T23:29:22.235230+0000","last_in_change":"2026-02-22T23:27:24.830980+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T23:27:34.171905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-22T23:27:43.179120+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6809","nonce":3878013816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6810","nonce":3878013816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6812","nonce":3878013816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6811","nonce":3878013816}]},"public_addr":"10.20.193.115:6809/3878013816","cluster_addr":"10.20.193.115:6810/3878013816","heartbeat_back_addr":"10.20.193.115:6812/3878013816","heartbeat_front_addr":"10.20.193.115:6811/3878013816","state":["exists","up"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":23,"up_thru":26,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6805","nonce":2409177103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6806","nonce":2409177103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6808","nonce":2409177103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6807","nonce":2409177103}]},"public_addr":"10.20.193.115:6805/2409177103","cluster_addr":"10.20.193.115:6806/2409177103","heartbeat_back_addr":"10.20.193.115:6808/2409177103","heartbeat_front_addr":"10.20.193.115:6807/2409177103","state":["exists","up"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6801","nonce":1978117969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6802","nonce":1978117969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6804","nonce":1978117969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6803","nonce":1978117969}]},"public_addr":"10.20.193.115:6801/1978117969","cluster_addr":"10.20.193.115:6802/1978117969","heartbeat_back_addr":"10.20.193.115:6804/1978117969","heartbeat_front_addr":"10.20.193.115:6803/1978117969","state":["exists","up"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":25,"up_from":29,"up_thru":29,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6813","nonce":1033269798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6814","nonce":1033269798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6816","nonce":1033269798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6815","nonce":1033269798}]},"public_addr":"10.20.193.115:6813/1033269798","cluster_addr":"10.20.193.115:6814/1033269798","heartbeat_back_addr":"10.20.193.115:6816/1033269798","heartbeat_front_addr":"10.20.193.115:6815/1033269798","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:32.314887+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:31.613759+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:31.523737+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:32.017476+0000","dead_epoch":26}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:29:25.238 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:osd.3 is up 2026-02-22T23:29:25.238 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2026-02-22T23:29:25.385 INFO:tasks.ceph.mon.a.trial115.stderr:2026-02-22T23:29:25.384+0000 7f2aeaf18640 -1 mon.a@0(leader).osd e30 definitely_dead 0 2026-02-22T23:29:26.247 INFO:teuthology.orchestra.run.trial115.stderr:marked out osd.1. 2026-02-22T23:29:26.262 DEBUG:tasks.ec_inconsistent_hinfo:waiting for pg 2.0 backfill from osd.1 to osd.0 complete 2026-02-22T23:29:31.263 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:29:31.263 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:29:31.264 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:29:31.264 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:29:31.354 INFO:teuthology.orchestra.run.trial115.stdout:38654705693 2026-02-22T23:29:31.354 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:29:31.355 INFO:teuthology.orchestra.run.trial115.stdout:98784247828 2026-02-22T23:29:31.355 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:29:31.356 INFO:teuthology.orchestra.run.trial115.stdout:124554051588 2026-02-22T23:29:31.356 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:29:31.358 INFO:teuthology.orchestra.run.trial115.stdout:42949672990 2026-02-22T23:29:31.358 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:29:31.508 INFO:teuthology.orchestra.run.trial115.stdout:38654705692 2026-02-22T23:29:31.522 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705693 got 38654705692 for osd.2 2026-02-22T23:29:31.554 INFO:teuthology.orchestra.run.trial115.stdout:124554051586 2026-02-22T23:29:31.558 INFO:teuthology.orchestra.run.trial115.stdout:98784247827 2026-02-22T23:29:31.566 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051588 got 124554051586 for osd.3 2026-02-22T23:29:31.567 INFO:teuthology.orchestra.run.trial115.stdout:42949672988 2026-02-22T23:29:31.571 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247828 got 98784247827 for osd.1 2026-02-22T23:29:31.581 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672990 got 42949672988 for osd.0 2026-02-22T23:29:32.523 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:29:32.567 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:29:32.572 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:29:32.582 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:29:32.683 INFO:teuthology.orchestra.run.trial115.stdout:38654705693 2026-02-22T23:29:32.697 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705693 got 38654705693 for osd.2 2026-02-22T23:29:32.697 DEBUG:teuthology.parallel:result is None 2026-02-22T23:29:32.721 INFO:teuthology.orchestra.run.trial115.stdout:98784247828 2026-02-22T23:29:32.733 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247828 got 98784247828 for osd.1 2026-02-22T23:29:32.734 DEBUG:teuthology.parallel:result is None 2026-02-22T23:29:32.739 INFO:teuthology.orchestra.run.trial115.stdout:124554051588 2026-02-22T23:29:32.754 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051588 got 124554051588 for osd.3 2026-02-22T23:29:32.755 DEBUG:teuthology.parallel:result is None 2026-02-22T23:29:32.768 INFO:teuthology.orchestra.run.trial115.stdout:42949672990 2026-02-22T23:29:32.781 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672990 got 42949672990 for osd.0 2026-02-22T23:29:32.781 DEBUG:teuthology.parallel:result is None 2026-02-22T23:29:32.781 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:29:32.979 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:29:32.979 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:29:32.991 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2026-02-22T23:29:32.149374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":66510,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2093,"num_bytes_recovered":17457851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":789624,"kb_used_data":557948,"kb_used_omap":7,"kb_used_meta":231608,"kb_avail":282325896,"statfs":{"total":289910292480,"available":289101717504,"internally_reserved":0,"allocated":571338752,"data_stored":571433452,"data_compressed":13032,"data_compressed_allocated":57344,"data_compressed_original":688128,"omap_allocated":7633,"internal_metadata":237167151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-68599,"num_objects_misplaced":66510,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2092,"num_bytes_recovered":17449472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000752"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":72866,"reported_epoch":33,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:29:31.346205+0000","last_change":"2026-02-22T23:29:28.357217+0000","last_active":"2026-02-22T23:29:31.346205+0000","last_peered":"2026-02-22T23:29:31.346205+0000","last_clean":"2026-02-22T23:29:23.243943+0000","last_became_active":"2026-02-22T23:29:28.356529+0000","last_became_peered":"2026-02-22T23:29:28.356529+0000","last_unstale":"2026-02-22T23:29:31.346205+0000","last_undegraded":"2026-02-22T23:29:31.346205+0000","last_fullsized":"2026-02-22T23:29:31.346205+0000","mapping_epoch":32,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":66510,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2091,"num_bytes_recovered":17129659,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":56,"reported_epoch":33,"state":"active+clean","last_fresh":"2026-02-22T23:29:28.342946+0000","last_change":"2026-02-22T23:29:27.256843+0000","last_active":"2026-02-22T23:29:28.342946+0000","last_peered":"2026-02-22T23:29:28.342946+0000","last_clean":"2026-02-22T23:29:28.342946+0000","last_became_active":"2026-02-22T23:29:27.253029+0000","last_became_peered":"2026-02-22T23:29:27.253029+0000","last_unstale":"2026-02-22T23:29:28.342946+0000","last_undegraded":"2026-02-22T23:29:28.342946+0000","last_fullsized":"2026-02-22T23:29:28.342946+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:36:17.698676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":66510,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2091,"num_bytes_recovered":17129659,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":851509248,"data_stored":851533824,"data_compressed":16108,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":655872,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949672990,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35552,"kb_used_data":8592,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94336288,"statfs":{"total":96636764160,"available":96600358912,"internally_reserved":0,"allocated":8798208,"data_stored":8924324,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051588,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313604,"kb_used_data":274668,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058236,"statfs":{"total":96636764160,"available":96315633664,"internally_reserved":0,"allocated":281260032,"data_stored":281090724,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440468,"kb_used_data":274688,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931372,"statfs":{"total":96636764160,"available":96185724928,"internally_reserved":0,"allocated":281280512,"data_stored":281418404,"data_compressed":7527,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":98784247828,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8568832,"data_stored":8568832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:29:32.992 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 72866, 'reported_epoch': 33, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:29:31.346205+0000', 'last_change': '2026-02-22T23:29:28.357217+0000', 'last_active': '2026-02-22T23:29:31.346205+0000', 'last_peered': '2026-02-22T23:29:31.346205+0000', 'last_clean': '2026-02-22T23:29:23.243943+0000', 'last_became_active': '2026-02-22T23:29:28.356529+0000', 'last_became_peered': '2026-02-22T23:29:28.356529+0000', 'last_unstale': '2026-02-22T23:29:31.346205+0000', 'last_undegraded': '2026-02-22T23:29:31.346205+0000', 'last_fullsized': '2026-02-22T23:29:31.346205+0000', 'mapping_epoch': 32, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 30, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 66510, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2091, 'num_bytes_recovered': 17129659, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:29:32.992 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:29:37.994 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:29:37.994 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:29:37.994 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:29:37.994 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:29:38.081 INFO:teuthology.orchestra.run.trial115.stdout:38654705695 2026-02-22T23:29:38.082 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:29:38.084 INFO:teuthology.orchestra.run.trial115.stdout:42949672992 2026-02-22T23:29:38.085 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:29:38.086 INFO:teuthology.orchestra.run.trial115.stdout:98784247830 2026-02-22T23:29:38.086 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:29:38.087 INFO:teuthology.orchestra.run.trial115.stdout:124554051590 2026-02-22T23:29:38.088 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:29:38.249 INFO:teuthology.orchestra.run.trial115.stdout:38654705695 2026-02-22T23:29:38.265 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705695 got 38654705695 for osd.2 2026-02-22T23:29:38.265 DEBUG:teuthology.parallel:result is None 2026-02-22T23:29:38.284 INFO:teuthology.orchestra.run.trial115.stdout:98784247830 2026-02-22T23:29:38.287 INFO:teuthology.orchestra.run.trial115.stdout:42949672992 2026-02-22T23:29:38.298 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247830 got 98784247830 for osd.1 2026-02-22T23:29:38.299 DEBUG:teuthology.parallel:result is None 2026-02-22T23:29:38.299 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672992 got 42949672992 for osd.0 2026-02-22T23:29:38.300 DEBUG:teuthology.parallel:result is None 2026-02-22T23:29:38.302 INFO:teuthology.orchestra.run.trial115.stdout:124554051590 2026-02-22T23:29:38.317 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051590 got 124554051590 for osd.3 2026-02-22T23:29:38.318 DEBUG:teuthology.parallel:result is None 2026-02-22T23:29:38.318 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:29:38.517 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:29:38.518 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:29:38.530 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2026-02-22T23:29:38.150113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":61901,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6702,"num_bytes_recovered":55214779,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":806268,"kb_used_data":574592,"kb_used_omap":7,"kb_used_meta":231608,"kb_avail":282309252,"statfs":{"total":289910292480,"available":289084674048,"internally_reserved":0,"allocated":588382208,"data_stored":588476908,"data_compressed":13032,"data_compressed_allocated":57344,"data_compressed_original":688128,"omap_allocated":7633,"internal_metadata":237167151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":61901,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6701,"num_bytes_recovered":55206400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001195"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":82084,"reported_epoch":33,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:29:38.076970+0000","last_change":"2026-02-22T23:29:28.357217+0000","last_active":"2026-02-22T23:29:38.076970+0000","last_peered":"2026-02-22T23:29:38.076970+0000","last_clean":"2026-02-22T23:29:23.243943+0000","last_became_active":"2026-02-22T23:29:28.356529+0000","last_became_peered":"2026-02-22T23:29:28.356529+0000","last_unstale":"2026-02-22T23:29:38.076970+0000","last_undegraded":"2026-02-22T23:29:38.076970+0000","last_fullsized":"2026-02-22T23:29:38.076970+0000","mapping_epoch":32,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":61901,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6700,"num_bytes_recovered":54886587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":56,"reported_epoch":33,"state":"active+clean","last_fresh":"2026-02-22T23:29:28.342946+0000","last_change":"2026-02-22T23:29:27.256843+0000","last_active":"2026-02-22T23:29:28.342946+0000","last_peered":"2026-02-22T23:29:28.342946+0000","last_clean":"2026-02-22T23:29:28.342946+0000","last_became_active":"2026-02-22T23:29:27.253029+0000","last_became_peered":"2026-02-22T23:29:27.253029+0000","last_unstale":"2026-02-22T23:29:28.342946+0000","last_undegraded":"2026-02-22T23:29:28.342946+0000","last_fullsized":"2026-02-22T23:29:28.342946+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:36:17.698676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":61901,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6700,"num_bytes_recovered":54886587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":870371328,"data_stored":870395904,"data_compressed":16108,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":655872,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949672992,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52196,"kb_used_data":25236,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94319644,"statfs":{"total":96636764160,"available":96583315456,"internally_reserved":0,"allocated":25841664,"data_stored":25967780,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313604,"kb_used_data":274668,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058236,"statfs":{"total":96636764160,"available":96315633664,"internally_reserved":0,"allocated":281260032,"data_stored":281090724,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440468,"kb_used_data":274688,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931372,"statfs":{"total":96636764160,"available":96185724928,"internally_reserved":0,"allocated":281280512,"data_stored":281418404,"data_compressed":7527,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":98784247830,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":27430912,"data_stored":27430912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:29:38.532 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 82084, 'reported_epoch': 33, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:29:38.076970+0000', 'last_change': '2026-02-22T23:29:28.357217+0000', 'last_active': '2026-02-22T23:29:38.076970+0000', 'last_peered': '2026-02-22T23:29:38.076970+0000', 'last_clean': '2026-02-22T23:29:23.243943+0000', 'last_became_active': '2026-02-22T23:29:28.356529+0000', 'last_became_peered': '2026-02-22T23:29:28.356529+0000', 'last_unstale': '2026-02-22T23:29:38.076970+0000', 'last_undegraded': '2026-02-22T23:29:38.076970+0000', 'last_fullsized': '2026-02-22T23:29:38.076970+0000', 'mapping_epoch': 32, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 30, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 61901, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6700, 'num_bytes_recovered': 54886587, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:29:38.532 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:29:43.534 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:29:43.534 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:29:43.535 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:29:43.535 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:29:43.623 INFO:teuthology.orchestra.run.trial115.stdout:124554051592 2026-02-22T23:29:43.624 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:29:43.624 INFO:teuthology.orchestra.run.trial115.stdout:38654705697 2026-02-22T23:29:43.625 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:29:43.626 INFO:teuthology.orchestra.run.trial115.stdout:42949672994 2026-02-22T23:29:43.626 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:29:43.629 INFO:teuthology.orchestra.run.trial115.stdout:98784247833 2026-02-22T23:29:43.629 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:29:43.823 INFO:teuthology.orchestra.run.trial115.stdout:124554051591 2026-02-22T23:29:43.826 INFO:teuthology.orchestra.run.trial115.stdout:38654705696 2026-02-22T23:29:43.837 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051592 got 124554051591 for osd.3 2026-02-22T23:29:43.840 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705697 got 38654705696 for osd.2 2026-02-22T23:29:43.840 INFO:teuthology.orchestra.run.trial115.stdout:42949672993 2026-02-22T23:29:43.854 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672994 got 42949672993 for osd.0 2026-02-22T23:29:43.866 INFO:teuthology.orchestra.run.trial115.stdout:98784247831 2026-02-22T23:29:43.880 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247833 got 98784247831 for osd.1 2026-02-22T23:29:44.837 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:29:44.841 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:29:44.855 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:29:44.880 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:29:44.990 INFO:teuthology.orchestra.run.trial115.stdout:124554051592 2026-02-22T23:29:45.003 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051592 got 124554051592 for osd.3 2026-02-22T23:29:45.004 DEBUG:teuthology.parallel:result is None 2026-02-22T23:29:45.028 INFO:teuthology.orchestra.run.trial115.stdout:42949672994 2026-02-22T23:29:45.029 INFO:teuthology.orchestra.run.trial115.stdout:38654705697 2026-02-22T23:29:45.041 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705697 got 38654705697 for osd.2 2026-02-22T23:29:45.042 DEBUG:teuthology.parallel:result is None 2026-02-22T23:29:45.042 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672994 got 42949672994 for osd.0 2026-02-22T23:29:45.042 DEBUG:teuthology.parallel:result is None 2026-02-22T23:29:45.048 INFO:teuthology.orchestra.run.trial115.stdout:98784247833 2026-02-22T23:29:45.062 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247833 got 98784247833 for osd.1 2026-02-22T23:29:45.062 DEBUG:teuthology.parallel:result is None 2026-02-22T23:29:45.062 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:29:45.259 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:29:45.260 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:29:45.271 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2026-02-22T23:29:44.150858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":57911,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10692,"num_bytes_recovered":87900859,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":859032,"kb_used_data":591260,"kb_used_omap":7,"kb_used_meta":267704,"kb_avail":282256488,"statfs":{"total":289910292480,"available":289030643712,"internally_reserved":0,"allocated":605450240,"data_stored":605544940,"data_compressed":13032,"data_compressed_allocated":57344,"data_compressed_original":688128,"omap_allocated":7633,"internal_metadata":274129455},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-8599,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8599,"num_bytes_recovered":70443008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001483"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":90064,"reported_epoch":33,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:29:43.613408+0000","last_change":"2026-02-22T23:29:28.357217+0000","last_active":"2026-02-22T23:29:43.613408+0000","last_peered":"2026-02-22T23:29:43.613408+0000","last_clean":"2026-02-22T23:29:23.243943+0000","last_became_active":"2026-02-22T23:29:28.356529+0000","last_became_peered":"2026-02-22T23:29:28.356529+0000","last_unstale":"2026-02-22T23:29:43.613408+0000","last_undegraded":"2026-02-22T23:29:43.613408+0000","last_fullsized":"2026-02-22T23:29:43.613408+0000","mapping_epoch":32,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":57911,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10690,"num_bytes_recovered":87572667,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":56,"reported_epoch":33,"state":"active+clean","last_fresh":"2026-02-22T23:29:28.342946+0000","last_change":"2026-02-22T23:29:27.256843+0000","last_active":"2026-02-22T23:29:28.342946+0000","last_peered":"2026-02-22T23:29:28.342946+0000","last_clean":"2026-02-22T23:29:28.342946+0000","last_became_active":"2026-02-22T23:29:27.253029+0000","last_became_peered":"2026-02-22T23:29:27.253029+0000","last_unstale":"2026-02-22T23:29:28.342946+0000","last_undegraded":"2026-02-22T23:29:28.342946+0000","last_fullsized":"2026-02-22T23:29:28.342946+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:36:17.698676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":57911,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10690,"num_bytes_recovered":87572667,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":886726656,"data_stored":886751232,"data_compressed":16108,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":655872,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949672994,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104960,"kb_used_data":41904,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94266880,"statfs":{"total":96636764160,"available":96529285120,"internally_reserved":0,"allocated":42909696,"data_stored":43035812,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051592,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313604,"kb_used_data":274668,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058236,"statfs":{"total":96636764160,"available":96315633664,"internally_reserved":0,"allocated":281260032,"data_stored":281090724,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440468,"kb_used_data":274688,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931372,"statfs":{"total":96636764160,"available":96185724928,"internally_reserved":0,"allocated":281280512,"data_stored":281418404,"data_compressed":7527,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":98784247833,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43786240,"data_stored":43786240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:29:45.272 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 90064, 'reported_epoch': 33, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:29:43.613408+0000', 'last_change': '2026-02-22T23:29:28.357217+0000', 'last_active': '2026-02-22T23:29:43.613408+0000', 'last_peered': '2026-02-22T23:29:43.613408+0000', 'last_clean': '2026-02-22T23:29:23.243943+0000', 'last_became_active': '2026-02-22T23:29:28.356529+0000', 'last_became_peered': '2026-02-22T23:29:28.356529+0000', 'last_unstale': '2026-02-22T23:29:43.613408+0000', 'last_undegraded': '2026-02-22T23:29:43.613408+0000', 'last_fullsized': '2026-02-22T23:29:43.613408+0000', 'mapping_epoch': 32, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 30, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 57911, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10690, 'num_bytes_recovered': 87572667, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:29:45.272 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:29:50.274 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:29:50.275 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:29:50.275 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:29:50.275 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:29:50.367 INFO:teuthology.orchestra.run.trial115.stdout:38654705700 2026-02-22T23:29:50.368 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:29:50.371 INFO:teuthology.orchestra.run.trial115.stdout:98784247835 2026-02-22T23:29:50.371 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:29:50.372 INFO:teuthology.orchestra.run.trial115.stdout:42949672997 2026-02-22T23:29:50.373 INFO:teuthology.orchestra.run.trial115.stdout:124554051594 2026-02-22T23:29:50.373 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:29:50.374 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:29:50.557 INFO:teuthology.orchestra.run.trial115.stdout:98784247834 2026-02-22T23:29:50.569 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247835 got 98784247834 for osd.1 2026-02-22T23:29:50.571 INFO:teuthology.orchestra.run.trial115.stdout:38654705699 2026-02-22T23:29:50.573 INFO:teuthology.orchestra.run.trial115.stdout:124554051593 2026-02-22T23:29:50.573 INFO:teuthology.orchestra.run.trial115.stdout:42949672995 2026-02-22T23:29:50.587 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705700 got 38654705699 for osd.2 2026-02-22T23:29:50.588 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672997 got 42949672995 for osd.0 2026-02-22T23:29:50.589 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051594 got 124554051593 for osd.3 2026-02-22T23:29:51.570 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:29:51.587 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:29:51.588 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:29:51.589 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:29:51.724 INFO:teuthology.orchestra.run.trial115.stdout:98784247834 2026-02-22T23:29:51.740 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247835 got 98784247834 for osd.1 2026-02-22T23:29:51.760 INFO:teuthology.orchestra.run.trial115.stdout:42949672995 2026-02-22T23:29:51.760 INFO:teuthology.orchestra.run.trial115.stdout:38654705699 2026-02-22T23:29:51.771 INFO:teuthology.orchestra.run.trial115.stdout:124554051593 2026-02-22T23:29:51.773 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705700 got 38654705699 for osd.2 2026-02-22T23:29:51.774 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672997 got 42949672995 for osd.0 2026-02-22T23:29:51.785 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051594 got 124554051593 for osd.3 2026-02-22T23:29:52.740 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:29:52.774 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:29:52.775 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:29:52.786 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:29:52.894 INFO:teuthology.orchestra.run.trial115.stdout:98784247835 2026-02-22T23:29:52.907 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247835 got 98784247835 for osd.1 2026-02-22T23:29:52.908 DEBUG:teuthology.parallel:result is None 2026-02-22T23:29:52.940 INFO:teuthology.orchestra.run.trial115.stdout:42949672997 2026-02-22T23:29:52.948 INFO:teuthology.orchestra.run.trial115.stdout:38654705700 2026-02-22T23:29:52.949 INFO:teuthology.orchestra.run.trial115.stdout:124554051595 2026-02-22T23:29:52.954 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672997 got 42949672997 for osd.0 2026-02-22T23:29:52.955 DEBUG:teuthology.parallel:result is None 2026-02-22T23:29:52.964 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705700 got 38654705700 for osd.2 2026-02-22T23:29:52.964 DEBUG:teuthology.parallel:result is None 2026-02-22T23:29:52.965 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051594 got 124554051595 for osd.3 2026-02-22T23:29:52.965 DEBUG:teuthology.parallel:result is None 2026-02-22T23:29:52.965 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:29:53.162 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:29:53.163 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:29:53.174 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2026-02-22T23:29:52.151846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":52511,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16092,"num_bytes_recovered":132137846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":923564,"kb_used_data":611376,"kb_used_omap":7,"kb_used_meta":312120,"kb_avail":282191956,"statfs":{"total":289910292480,"available":288964562944,"internally_reserved":0,"allocated":626049024,"data_stored":626147820,"data_compressed":15163,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":7633,"internal_metadata":319611439},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-9390,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9390,"num_bytes_recovered":76923067,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001494"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":100864,"reported_epoch":33,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:29:51.240375+0000","last_change":"2026-02-22T23:29:28.357217+0000","last_active":"2026-02-22T23:29:51.240375+0000","last_peered":"2026-02-22T23:29:51.240375+0000","last_clean":"2026-02-22T23:29:23.243943+0000","last_became_active":"2026-02-22T23:29:28.356529+0000","last_became_peered":"2026-02-22T23:29:28.356529+0000","last_unstale":"2026-02-22T23:29:51.240375+0000","last_undegraded":"2026-02-22T23:29:51.240375+0000","last_fullsized":"2026-02-22T23:29:51.240375+0000","mapping_epoch":32,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":52511,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16090,"num_bytes_recovered":131809654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":56,"reported_epoch":33,"state":"active+clean","last_fresh":"2026-02-22T23:29:28.342946+0000","last_change":"2026-02-22T23:29:27.256843+0000","last_active":"2026-02-22T23:29:28.342946+0000","last_peered":"2026-02-22T23:29:28.342946+0000","last_clean":"2026-02-22T23:29:28.342946+0000","last_became_active":"2026-02-22T23:29:27.253029+0000","last_became_peered":"2026-02-22T23:29:27.253029+0000","last_unstale":"2026-02-22T23:29:28.342946+0000","last_undegraded":"2026-02-22T23:29:28.342946+0000","last_fullsized":"2026-02-22T23:29:28.342946+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:36:17.698676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":52511,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16090,"num_bytes_recovered":131809654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":906260480,"data_stored":906289152,"data_compressed":18239,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":655872,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949672997,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":169492,"kb_used_data":62020,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":94202348,"statfs":{"total":96636764160,"available":96463204352,"internally_reserved":0,"allocated":63508480,"data_stored":63638692,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051595,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313604,"kb_used_data":274668,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058236,"statfs":{"total":96636764160,"available":96315633664,"internally_reserved":0,"allocated":281260032,"data_stored":281090724,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440468,"kb_used_data":274688,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931372,"statfs":{"total":96636764160,"available":96185724928,"internally_reserved":0,"allocated":281280512,"data_stored":281418404,"data_compressed":7527,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":98784247835,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":63320064,"data_stored":63324160,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:29:53.175 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 100864, 'reported_epoch': 33, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:29:51.240375+0000', 'last_change': '2026-02-22T23:29:28.357217+0000', 'last_active': '2026-02-22T23:29:51.240375+0000', 'last_peered': '2026-02-22T23:29:51.240375+0000', 'last_clean': '2026-02-22T23:29:23.243943+0000', 'last_became_active': '2026-02-22T23:29:28.356529+0000', 'last_became_peered': '2026-02-22T23:29:28.356529+0000', 'last_unstale': '2026-02-22T23:29:51.240375+0000', 'last_undegraded': '2026-02-22T23:29:51.240375+0000', 'last_fullsized': '2026-02-22T23:29:51.240375+0000', 'mapping_epoch': 32, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 30, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 52511, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16090, 'num_bytes_recovered': 131809654, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:29:53.176 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:29:58.177 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:29:58.177 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:29:58.178 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:29:58.178 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:29:58.268 INFO:teuthology.orchestra.run.trial115.stdout:124554051597 2026-02-22T23:29:58.269 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:29:58.270 INFO:teuthology.orchestra.run.trial115.stdout:38654705702 2026-02-22T23:29:58.271 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:29:58.272 INFO:teuthology.orchestra.run.trial115.stdout:42949672999 2026-02-22T23:29:58.272 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:29:58.273 INFO:teuthology.orchestra.run.trial115.stdout:98784247838 2026-02-22T23:29:58.274 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:29:58.465 INFO:teuthology.orchestra.run.trial115.stdout:38654705701 2026-02-22T23:29:58.473 INFO:teuthology.orchestra.run.trial115.stdout:124554051596 2026-02-22T23:29:58.477 INFO:teuthology.orchestra.run.trial115.stdout:98784247836 2026-02-22T23:29:58.478 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705702 got 38654705701 for osd.2 2026-02-22T23:29:58.485 INFO:teuthology.orchestra.run.trial115.stdout:42949672998 2026-02-22T23:29:58.486 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051597 got 124554051596 for osd.3 2026-02-22T23:29:58.491 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247838 got 98784247836 for osd.1 2026-02-22T23:29:58.499 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672999 got 42949672998 for osd.0 2026-02-22T23:29:59.479 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:29:59.486 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:29:59.491 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:29:59.499 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:29:59.628 INFO:teuthology.orchestra.run.trial115.stdout:38654705701 2026-02-22T23:29:59.641 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705702 got 38654705701 for osd.2 2026-02-22T23:29:59.678 INFO:teuthology.orchestra.run.trial115.stdout:42949672998 2026-02-22T23:29:59.686 INFO:teuthology.orchestra.run.trial115.stdout:98784247836 2026-02-22T23:29:59.692 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672999 got 42949672998 for osd.0 2026-02-22T23:29:59.694 INFO:teuthology.orchestra.run.trial115.stdout:124554051596 2026-02-22T23:29:59.699 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247838 got 98784247836 for osd.1 2026-02-22T23:29:59.709 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051597 got 124554051596 for osd.3 2026-02-22T23:30:00.643 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:00.693 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:00.700 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:00.709 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:00.801 INFO:teuthology.orchestra.run.trial115.stdout:38654705703 2026-02-22T23:30:00.815 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705702 got 38654705703 for osd.2 2026-02-22T23:30:00.815 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:00.846 INFO:teuthology.orchestra.run.trial115.stdout:42949672999 2026-02-22T23:30:00.860 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672999 got 42949672999 for osd.0 2026-02-22T23:30:00.860 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:00.888 INFO:teuthology.orchestra.run.trial115.stdout:98784247838 2026-02-22T23:30:00.897 INFO:teuthology.orchestra.run.trial115.stdout:124554051597 2026-02-22T23:30:00.901 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247838 got 98784247838 for osd.1 2026-02-22T23:30:00.902 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:00.912 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051597 got 124554051597 for osd.3 2026-02-22T23:30:00.912 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:00.912 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:30:01.105 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:30:01.106 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:30:01.117 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2026-02-22T23:30:00.152887+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":47629,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20974,"num_bytes_recovered":172131190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":934896,"kb_used_data":631028,"kb_used_omap":7,"kb_used_meta":303800,"kb_avail":282180624,"statfs":{"total":289910292480,"available":288952958976,"internally_reserved":0,"allocated":646172672,"data_stored":646271468,"data_compressed":15163,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":7633,"internal_metadata":311091759},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-8414,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8414,"num_bytes_recovered":68927488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001544"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":110628,"reported_epoch":33,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:29:58.259294+0000","last_change":"2026-02-22T23:29:28.357217+0000","last_active":"2026-02-22T23:29:58.259294+0000","last_peered":"2026-02-22T23:29:58.259294+0000","last_clean":"2026-02-22T23:29:23.243943+0000","last_became_active":"2026-02-22T23:29:28.356529+0000","last_became_peered":"2026-02-22T23:29:28.356529+0000","last_unstale":"2026-02-22T23:29:58.259294+0000","last_undegraded":"2026-02-22T23:29:58.259294+0000","last_fullsized":"2026-02-22T23:29:58.259294+0000","mapping_epoch":32,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":47629,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20972,"num_bytes_recovered":171802998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":57,"reported_epoch":33,"state":"active+clean","last_fresh":"2026-02-22T23:29:28.342971+0000","last_change":"2026-02-22T23:29:27.256843+0000","last_active":"2026-02-22T23:29:28.342971+0000","last_peered":"2026-02-22T23:29:28.342971+0000","last_clean":"2026-02-22T23:29:28.342971+0000","last_became_active":"2026-02-22T23:29:27.253029+0000","last_became_peered":"2026-02-22T23:29:27.253029+0000","last_unstale":"2026-02-22T23:29:28.342971+0000","last_undegraded":"2026-02-22T23:29:28.342971+0000","last_fullsized":"2026-02-22T23:29:28.342971+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:36:17.698676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":47629,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20972,"num_bytes_recovered":171802998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":928841728,"data_stored":928870400,"data_compressed":18239,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":655872,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949672999,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":180824,"kb_used_data":81672,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":94191016,"statfs":{"total":96636764160,"available":96451600384,"internally_reserved":0,"allocated":83632128,"data_stored":83762340,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":1594,"internal_metadata":101513670},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051597,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313604,"kb_used_data":274668,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058236,"statfs":{"total":96636764160,"available":96315633664,"internally_reserved":0,"allocated":281260032,"data_stored":281090724,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440468,"kb_used_data":274688,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931372,"statfs":{"total":96636764160,"available":96185724928,"internally_reserved":0,"allocated":281280512,"data_stored":281418404,"data_compressed":7527,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":98784247838,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":85901312,"data_stored":85905408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:30:01.118 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 110628, 'reported_epoch': 33, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:29:58.259294+0000', 'last_change': '2026-02-22T23:29:28.357217+0000', 'last_active': '2026-02-22T23:29:58.259294+0000', 'last_peered': '2026-02-22T23:29:58.259294+0000', 'last_clean': '2026-02-22T23:29:23.243943+0000', 'last_became_active': '2026-02-22T23:29:28.356529+0000', 'last_became_peered': '2026-02-22T23:29:28.356529+0000', 'last_unstale': '2026-02-22T23:29:58.259294+0000', 'last_undegraded': '2026-02-22T23:29:58.259294+0000', 'last_fullsized': '2026-02-22T23:29:58.259294+0000', 'mapping_epoch': 32, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 30, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 47629, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20972, 'num_bytes_recovered': 171802998, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:30:01.118 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:30:06.119 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:30:06.120 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:30:06.121 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:30:06.121 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:30:06.212 INFO:teuthology.orchestra.run.trial115.stdout:42949673002 2026-02-22T23:30:06.213 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:06.214 INFO:teuthology.orchestra.run.trial115.stdout:124554051599 2026-02-22T23:30:06.215 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:06.215 INFO:teuthology.orchestra.run.trial115.stdout:98784247840 2026-02-22T23:30:06.216 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:06.217 INFO:teuthology.orchestra.run.trial115.stdout:38654705705 2026-02-22T23:30:06.218 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:06.409 INFO:teuthology.orchestra.run.trial115.stdout:38654705704 2026-02-22T23:30:06.411 INFO:teuthology.orchestra.run.trial115.stdout:98784247839 2026-02-22T23:30:06.413 INFO:teuthology.orchestra.run.trial115.stdout:42949673001 2026-02-22T23:30:06.415 INFO:teuthology.orchestra.run.trial115.stdout:124554051598 2026-02-22T23:30:06.424 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705705 got 38654705704 for osd.2 2026-02-22T23:30:06.425 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247840 got 98784247839 for osd.1 2026-02-22T23:30:06.427 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673002 got 42949673001 for osd.0 2026-02-22T23:30:06.429 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051599 got 124554051598 for osd.3 2026-02-22T23:30:07.424 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:07.426 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:07.427 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:07.430 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:07.572 INFO:teuthology.orchestra.run.trial115.stdout:38654705704 2026-02-22T23:30:07.574 INFO:teuthology.orchestra.run.trial115.stdout:98784247839 2026-02-22T23:30:07.586 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705705 got 38654705704 for osd.2 2026-02-22T23:30:07.587 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247840 got 98784247839 for osd.1 2026-02-22T23:30:07.631 INFO:teuthology.orchestra.run.trial115.stdout:42949673001 2026-02-22T23:30:07.634 INFO:teuthology.orchestra.run.trial115.stdout:124554051598 2026-02-22T23:30:07.645 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673002 got 42949673001 for osd.0 2026-02-22T23:30:07.648 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051599 got 124554051598 for osd.3 2026-02-22T23:30:08.587 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:08.588 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:08.646 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:08.648 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:08.752 INFO:teuthology.orchestra.run.trial115.stdout:98784247840 2026-02-22T23:30:08.759 INFO:teuthology.orchestra.run.trial115.stdout:38654705705 2026-02-22T23:30:08.764 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247840 got 98784247840 for osd.1 2026-02-22T23:30:08.765 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:08.774 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705705 got 38654705705 for osd.2 2026-02-22T23:30:08.774 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:08.799 INFO:teuthology.orchestra.run.trial115.stdout:124554051600 2026-02-22T23:30:08.803 INFO:teuthology.orchestra.run.trial115.stdout:42949673002 2026-02-22T23:30:08.811 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051599 got 124554051600 for osd.3 2026-02-22T23:30:08.811 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:08.815 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673002 got 42949673002 for osd.0 2026-02-22T23:30:08.816 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:08.816 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:30:09.024 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:30:09.025 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:30:09.038 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2026-02-22T23:30:08.153880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":41971,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26632,"num_bytes_recovered":218481526,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1006344,"kb_used_data":653900,"kb_used_omap":7,"kb_used_meta":352376,"kb_avail":282109176,"statfs":{"total":289910292480,"available":288879796224,"internally_reserved":0,"allocated":669593600,"data_stored":669692396,"data_compressed":15163,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":7633,"internal_metadata":360833583},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-10540,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10540,"num_bytes_recovered":86343680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001522"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":121944,"reported_epoch":33,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:30:06.241978+0000","last_change":"2026-02-22T23:29:28.357217+0000","last_active":"2026-02-22T23:30:06.241978+0000","last_peered":"2026-02-22T23:30:06.241978+0000","last_clean":"2026-02-22T23:29:23.243943+0000","last_became_active":"2026-02-22T23:29:28.356529+0000","last_became_peered":"2026-02-22T23:29:28.356529+0000","last_unstale":"2026-02-22T23:30:06.241978+0000","last_undegraded":"2026-02-22T23:30:06.241978+0000","last_fullsized":"2026-02-22T23:30:06.241978+0000","mapping_epoch":32,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":41971,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26630,"num_bytes_recovered":218153334,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":57,"reported_epoch":33,"state":"active+clean","last_fresh":"2026-02-22T23:29:28.342971+0000","last_change":"2026-02-22T23:29:27.256843+0000","last_active":"2026-02-22T23:29:28.342971+0000","last_peered":"2026-02-22T23:29:28.342971+0000","last_clean":"2026-02-22T23:29:28.342971+0000","last_became_active":"2026-02-22T23:29:27.253029+0000","last_became_peered":"2026-02-22T23:29:27.253029+0000","last_unstale":"2026-02-22T23:29:28.342971+0000","last_undegraded":"2026-02-22T23:29:28.342971+0000","last_fullsized":"2026-02-22T23:29:28.342971+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:36:17.698676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":41971,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26630,"num_bytes_recovered":218153334,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":951930880,"data_stored":951959552,"data_compressed":18239,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":655872,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673002,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":252272,"kb_used_data":104544,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94119568,"statfs":{"total":96636764160,"available":96378437632,"internally_reserved":0,"allocated":107053056,"data_stored":107183268,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":1594,"internal_metadata":151255494},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051600,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313604,"kb_used_data":274668,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058236,"statfs":{"total":96636764160,"available":96315633664,"internally_reserved":0,"allocated":281260032,"data_stored":281090724,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440468,"kb_used_data":274688,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931372,"statfs":{"total":96636764160,"available":96185724928,"internally_reserved":0,"allocated":281280512,"data_stored":281418404,"data_compressed":7527,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":98784247840,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":108990464,"data_stored":108994560,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:30:09.039 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 121944, 'reported_epoch': 33, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:30:06.241978+0000', 'last_change': '2026-02-22T23:29:28.357217+0000', 'last_active': '2026-02-22T23:30:06.241978+0000', 'last_peered': '2026-02-22T23:30:06.241978+0000', 'last_clean': '2026-02-22T23:29:23.243943+0000', 'last_became_active': '2026-02-22T23:29:28.356529+0000', 'last_became_peered': '2026-02-22T23:29:28.356529+0000', 'last_unstale': '2026-02-22T23:30:06.241978+0000', 'last_undegraded': '2026-02-22T23:30:06.241978+0000', 'last_fullsized': '2026-02-22T23:30:06.241978+0000', 'mapping_epoch': 32, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 30, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 41971, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26630, 'num_bytes_recovered': 218153334, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:30:09.039 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:30:14.041 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:30:14.042 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:30:14.042 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:30:14.043 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:30:14.132 INFO:teuthology.orchestra.run.trial115.stdout:98784247843 2026-02-22T23:30:14.133 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:14.134 INFO:teuthology.orchestra.run.trial115.stdout:124554051602 2026-02-22T23:30:14.134 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:14.136 INFO:teuthology.orchestra.run.trial115.stdout:38654705707 2026-02-22T23:30:14.137 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:14.138 INFO:teuthology.orchestra.run.trial115.stdout:42949673004 2026-02-22T23:30:14.138 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:14.337 INFO:teuthology.orchestra.run.trial115.stdout:98784247843 2026-02-22T23:30:14.341 INFO:teuthology.orchestra.run.trial115.stdout:124554051602 2026-02-22T23:30:14.352 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247843 got 98784247843 for osd.1 2026-02-22T23:30:14.353 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:14.356 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051602 got 124554051602 for osd.3 2026-02-22T23:30:14.356 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:14.376 INFO:teuthology.orchestra.run.trial115.stdout:38654705707 2026-02-22T23:30:14.381 INFO:teuthology.orchestra.run.trial115.stdout:42949673004 2026-02-22T23:30:14.388 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705707 got 38654705707 for osd.2 2026-02-22T23:30:14.389 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:14.395 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673004 got 42949673004 for osd.0 2026-02-22T23:30:14.396 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:14.396 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:30:14.593 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:30:14.593 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:30:14.606 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2026-02-22T23:30:14.154668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":36383,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32220,"num_bytes_recovered":264258422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1061260,"kb_used_data":676880,"kb_used_omap":7,"kb_used_meta":384312,"kb_avail":282054260,"statfs":{"total":289910292480,"available":288823562240,"internally_reserved":0,"allocated":693125120,"data_stored":693223916,"data_compressed":15163,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":7633,"internal_metadata":393536047},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-9125,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9125,"num_bytes_recovered":74752000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001538"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":133120,"reported_epoch":33,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:30:14.123672+0000","last_change":"2026-02-22T23:29:28.357217+0000","last_active":"2026-02-22T23:30:14.123672+0000","last_peered":"2026-02-22T23:30:14.123672+0000","last_clean":"2026-02-22T23:29:23.243943+0000","last_became_active":"2026-02-22T23:29:28.356529+0000","last_became_peered":"2026-02-22T23:29:28.356529+0000","last_unstale":"2026-02-22T23:30:14.123672+0000","last_undegraded":"2026-02-22T23:30:14.123672+0000","last_fullsized":"2026-02-22T23:30:14.123672+0000","mapping_epoch":32,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":36383,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32218,"num_bytes_recovered":263930230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":57,"reported_epoch":33,"state":"active+clean","last_fresh":"2026-02-22T23:29:28.342971+0000","last_change":"2026-02-22T23:29:27.256843+0000","last_active":"2026-02-22T23:29:28.342971+0000","last_peered":"2026-02-22T23:29:28.342971+0000","last_clean":"2026-02-22T23:29:28.342971+0000","last_became_active":"2026-02-22T23:29:27.253029+0000","last_became_peered":"2026-02-22T23:29:27.253029+0000","last_unstale":"2026-02-22T23:29:28.342971+0000","last_undegraded":"2026-02-22T23:29:28.342971+0000","last_fullsized":"2026-02-22T23:29:28.342971+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:36:17.698676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":36383,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32218,"num_bytes_recovered":263930230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974913536,"data_stored":974942208,"data_compressed":18239,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":655872,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673004,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307188,"kb_used_data":127524,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":94064652,"statfs":{"total":96636764160,"available":96322203648,"internally_reserved":0,"allocated":130584576,"data_stored":130714788,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051602,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313604,"kb_used_data":274668,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058236,"statfs":{"total":96636764160,"available":96315633664,"internally_reserved":0,"allocated":281260032,"data_stored":281090724,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440468,"kb_used_data":274688,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931372,"statfs":{"total":96636764160,"available":96185724928,"internally_reserved":0,"allocated":281280512,"data_stored":281418404,"data_compressed":7527,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":98784247843,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":131973120,"data_stored":131977216,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:30:14.607 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 133120, 'reported_epoch': 33, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:30:14.123672+0000', 'last_change': '2026-02-22T23:29:28.357217+0000', 'last_active': '2026-02-22T23:30:14.123672+0000', 'last_peered': '2026-02-22T23:30:14.123672+0000', 'last_clean': '2026-02-22T23:29:23.243943+0000', 'last_became_active': '2026-02-22T23:29:28.356529+0000', 'last_became_peered': '2026-02-22T23:29:28.356529+0000', 'last_unstale': '2026-02-22T23:30:14.123672+0000', 'last_undegraded': '2026-02-22T23:30:14.123672+0000', 'last_fullsized': '2026-02-22T23:30:14.123672+0000', 'mapping_epoch': 32, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 30, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 36383, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32218, 'num_bytes_recovered': 263930230, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:30:14.608 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:30:19.609 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:30:19.610 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:30:19.610 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:30:19.611 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:30:19.699 INFO:teuthology.orchestra.run.trial115.stdout:42949673006 2026-02-22T23:30:19.699 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:19.700 INFO:teuthology.orchestra.run.trial115.stdout:124554051604 2026-02-22T23:30:19.701 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:19.702 INFO:teuthology.orchestra.run.trial115.stdout:98784247845 2026-02-22T23:30:19.702 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:19.704 INFO:teuthology.orchestra.run.trial115.stdout:38654705710 2026-02-22T23:30:19.704 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:19.897 INFO:teuthology.orchestra.run.trial115.stdout:42949673005 2026-02-22T23:30:19.902 INFO:teuthology.orchestra.run.trial115.stdout:98784247843 2026-02-22T23:30:19.909 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673006 got 42949673005 for osd.0 2026-02-22T23:30:19.913 INFO:teuthology.orchestra.run.trial115.stdout:124554051603 2026-02-22T23:30:19.914 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247845 got 98784247843 for osd.1 2026-02-22T23:30:19.928 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051604 got 124554051603 for osd.3 2026-02-22T23:30:19.947 INFO:teuthology.orchestra.run.trial115.stdout:38654705708 2026-02-22T23:30:19.962 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705710 got 38654705708 for osd.2 2026-02-22T23:30:20.910 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:20.914 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:20.929 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:20.962 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:21.066 INFO:teuthology.orchestra.run.trial115.stdout:42949673006 2026-02-22T23:30:21.079 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673006 got 42949673006 for osd.0 2026-02-22T23:30:21.080 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:21.119 INFO:teuthology.orchestra.run.trial115.stdout:98784247845 2026-02-22T23:30:21.124 INFO:teuthology.orchestra.run.trial115.stdout:124554051604 2026-02-22T23:30:21.133 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247845 got 98784247845 for osd.1 2026-02-22T23:30:21.133 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:21.137 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051604 got 124554051604 for osd.3 2026-02-22T23:30:21.138 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:21.167 INFO:teuthology.orchestra.run.trial115.stdout:38654705710 2026-02-22T23:30:21.180 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705710 got 38654705710 for osd.2 2026-02-22T23:30:21.180 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:21.181 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:30:21.375 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:30:21.376 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:30:21.387 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2026-02-22T23:30:20.155426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":32616,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35987,"num_bytes_recovered":295117686,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1095640,"kb_used_data":693212,"kb_used_omap":7,"kb_used_meta":402360,"kb_avail":282019880,"statfs":{"total":289910292480,"available":288788357120,"internally_reserved":0,"allocated":709849088,"data_stored":709947884,"data_compressed":15163,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":7633,"internal_metadata":412017199},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-9355,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9355,"num_bytes_recovered":76636160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001545"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":140654,"reported_epoch":33,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:30:19.691409+0000","last_change":"2026-02-22T23:29:28.357217+0000","last_active":"2026-02-22T23:30:19.691409+0000","last_peered":"2026-02-22T23:30:19.691409+0000","last_clean":"2026-02-22T23:29:23.243943+0000","last_became_active":"2026-02-22T23:29:28.356529+0000","last_became_peered":"2026-02-22T23:29:28.356529+0000","last_unstale":"2026-02-22T23:30:19.691409+0000","last_undegraded":"2026-02-22T23:30:19.691409+0000","last_fullsized":"2026-02-22T23:30:19.691409+0000","mapping_epoch":32,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":32616,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35985,"num_bytes_recovered":294789494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":57,"reported_epoch":33,"state":"active+clean","last_fresh":"2026-02-22T23:29:28.342971+0000","last_change":"2026-02-22T23:29:27.256843+0000","last_active":"2026-02-22T23:29:28.342971+0000","last_peered":"2026-02-22T23:29:28.342971+0000","last_clean":"2026-02-22T23:29:28.342971+0000","last_became_active":"2026-02-22T23:29:27.253029+0000","last_became_peered":"2026-02-22T23:29:27.253029+0000","last_unstale":"2026-02-22T23:29:28.342971+0000","last_undegraded":"2026-02-22T23:29:28.342971+0000","last_fullsized":"2026-02-22T23:29:28.342971+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:36:17.698676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":32616,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35985,"num_bytes_recovered":294789494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":990330880,"data_stored":990359552,"data_compressed":18239,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":655872,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673006,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":341568,"kb_used_data":143856,"kb_used_omap":1,"kb_used_meta":197694,"kb_avail":94030272,"statfs":{"total":96636764160,"available":96286998528,"internally_reserved":0,"allocated":147308544,"data_stored":147438756,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":1594,"internal_metadata":202439110},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051604,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313604,"kb_used_data":274668,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058236,"statfs":{"total":96636764160,"available":96315633664,"internally_reserved":0,"allocated":281260032,"data_stored":281090724,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440468,"kb_used_data":274688,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931372,"statfs":{"total":96636764160,"available":96185724928,"internally_reserved":0,"allocated":281280512,"data_stored":281418404,"data_compressed":7527,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":98784247845,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":147390464,"data_stored":147394560,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:30:21.388 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 140654, 'reported_epoch': 33, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:30:19.691409+0000', 'last_change': '2026-02-22T23:29:28.357217+0000', 'last_active': '2026-02-22T23:30:19.691409+0000', 'last_peered': '2026-02-22T23:30:19.691409+0000', 'last_clean': '2026-02-22T23:29:23.243943+0000', 'last_became_active': '2026-02-22T23:29:28.356529+0000', 'last_became_peered': '2026-02-22T23:29:28.356529+0000', 'last_unstale': '2026-02-22T23:30:19.691409+0000', 'last_undegraded': '2026-02-22T23:30:19.691409+0000', 'last_fullsized': '2026-02-22T23:30:19.691409+0000', 'mapping_epoch': 32, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 30, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 32616, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35985, 'num_bytes_recovered': 294789494, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:30:21.389 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:30:26.391 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:30:26.391 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:30:26.392 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:30:26.393 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:30:26.479 INFO:teuthology.orchestra.run.trial115.stdout:124554051607 2026-02-22T23:30:26.479 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:26.480 INFO:teuthology.orchestra.run.trial115.stdout:38654705712 2026-02-22T23:30:26.481 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:26.485 INFO:teuthology.orchestra.run.trial115.stdout:42949673009 2026-02-22T23:30:26.485 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:26.487 INFO:teuthology.orchestra.run.trial115.stdout:98784247847 2026-02-22T23:30:26.487 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:26.649 INFO:teuthology.orchestra.run.trial115.stdout:124554051605 2026-02-22T23:30:26.651 INFO:teuthology.orchestra.run.trial115.stdout:38654705711 2026-02-22T23:30:26.662 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051607 got 124554051605 for osd.3 2026-02-22T23:30:26.664 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705712 got 38654705711 for osd.2 2026-02-22T23:30:26.684 INFO:teuthology.orchestra.run.trial115.stdout:98784247846 2026-02-22T23:30:26.691 INFO:teuthology.orchestra.run.trial115.stdout:42949673008 2026-02-22T23:30:26.696 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247847 got 98784247846 for osd.1 2026-02-22T23:30:26.706 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673009 got 42949673008 for osd.0 2026-02-22T23:30:27.662 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:27.664 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:27.697 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:27.707 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:27.823 INFO:teuthology.orchestra.run.trial115.stdout:124554051605 2026-02-22T23:30:27.834 INFO:teuthology.orchestra.run.trial115.stdout:38654705711 2026-02-22T23:30:27.836 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051607 got 124554051605 for osd.3 2026-02-22T23:30:27.850 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705712 got 38654705711 for osd.2 2026-02-22T23:30:27.869 INFO:teuthology.orchestra.run.trial115.stdout:42949673008 2026-02-22T23:30:27.876 INFO:teuthology.orchestra.run.trial115.stdout:98784247846 2026-02-22T23:30:27.884 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673009 got 42949673008 for osd.0 2026-02-22T23:30:27.890 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247847 got 98784247846 for osd.1 2026-02-22T23:30:28.838 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:28.851 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:28.885 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:28.891 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:28.994 INFO:teuthology.orchestra.run.trial115.stdout:124554051607 2026-02-22T23:30:29.008 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051607 got 124554051607 for osd.3 2026-02-22T23:30:29.009 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:29.041 INFO:teuthology.orchestra.run.trial115.stdout:38654705712 2026-02-22T23:30:29.056 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705712 got 38654705712 for osd.2 2026-02-22T23:30:29.056 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:29.078 INFO:teuthology.orchestra.run.trial115.stdout:42949673009 2026-02-22T23:30:29.092 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673009 got 42949673009 for osd.0 2026-02-22T23:30:29.092 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:29.094 INFO:teuthology.orchestra.run.trial115.stdout:98784247847 2026-02-22T23:30:29.107 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247847 got 98784247847 for osd.1 2026-02-22T23:30:29.108 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:29.108 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:30:29.298 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:30:29.299 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:30:29.311 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2026-02-22T23:30:28.156384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27703,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40900,"num_bytes_recovered":335364982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1135272,"kb_used_data":710636,"kb_used_omap":7,"kb_used_meta":424568,"kb_avail":281980248,"statfs":{"total":289910292480,"available":288747773952,"internally_reserved":0,"allocated":727691264,"data_stored":727790060,"data_compressed":15163,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":7633,"internal_metadata":434758191},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-8680,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8680,"num_bytes_recovered":71106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001448"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":150480,"reported_epoch":33,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:30:26.471420+0000","last_change":"2026-02-22T23:29:28.357217+0000","last_active":"2026-02-22T23:30:26.471420+0000","last_peered":"2026-02-22T23:30:26.471420+0000","last_clean":"2026-02-22T23:29:23.243943+0000","last_became_active":"2026-02-22T23:29:28.356529+0000","last_became_peered":"2026-02-22T23:29:28.356529+0000","last_unstale":"2026-02-22T23:30:26.471420+0000","last_undegraded":"2026-02-22T23:30:26.471420+0000","last_fullsized":"2026-02-22T23:30:26.471420+0000","mapping_epoch":32,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27703,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40898,"num_bytes_recovered":335036790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":57,"reported_epoch":33,"state":"active+clean","last_fresh":"2026-02-22T23:29:28.342971+0000","last_change":"2026-02-22T23:29:27.256843+0000","last_active":"2026-02-22T23:29:28.342971+0000","last_peered":"2026-02-22T23:29:28.342971+0000","last_clean":"2026-02-22T23:29:28.342971+0000","last_became_active":"2026-02-22T23:29:27.253029+0000","last_became_peered":"2026-02-22T23:29:27.253029+0000","last_unstale":"2026-02-22T23:29:28.342971+0000","last_undegraded":"2026-02-22T23:29:28.342971+0000","last_fullsized":"2026-02-22T23:29:28.342971+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:36:17.698676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27703,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40898,"num_bytes_recovered":335036790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1010470912,"data_stored":1010499584,"data_compressed":18239,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":655872,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673009,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381200,"kb_used_data":161280,"kb_used_omap":1,"kb_used_meta":219902,"kb_avail":93990640,"statfs":{"total":96636764160,"available":96246415360,"internally_reserved":0,"allocated":165150720,"data_stored":165280932,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":1594,"internal_metadata":225180102},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051607,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313604,"kb_used_data":274668,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058236,"statfs":{"total":96636764160,"available":96315633664,"internally_reserved":0,"allocated":281260032,"data_stored":281090724,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440468,"kb_used_data":274688,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931372,"statfs":{"total":96636764160,"available":96185724928,"internally_reserved":0,"allocated":281280512,"data_stored":281418404,"data_compressed":7527,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":98784247847,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167530496,"data_stored":167534592,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:30:29.312 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 150480, 'reported_epoch': 33, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:30:26.471420+0000', 'last_change': '2026-02-22T23:29:28.357217+0000', 'last_active': '2026-02-22T23:30:26.471420+0000', 'last_peered': '2026-02-22T23:30:26.471420+0000', 'last_clean': '2026-02-22T23:29:23.243943+0000', 'last_became_active': '2026-02-22T23:29:28.356529+0000', 'last_became_peered': '2026-02-22T23:29:28.356529+0000', 'last_unstale': '2026-02-22T23:30:26.471420+0000', 'last_undegraded': '2026-02-22T23:30:26.471420+0000', 'last_fullsized': '2026-02-22T23:30:26.471420+0000', 'mapping_epoch': 32, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 30, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 27703, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 40898, 'num_bytes_recovered': 335036790, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:30:29.312 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:30:34.227 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:30:34.229+0000 7fd80d87f640 -1 osd.3 pg_epoch: 33 pg[2.0s0( v 25'68599 (25'68571,25'68599] local-lis/les=32/33 n=68599 ec=16/16 lis/c=32/29 les/c/f=33/30/0 sis=32) [3,0,2]/[3,1,2]p3(0) backfill=[0(1)] r=0 lpr=32 pi=[29,32)/1 crt=25'68599 lcod 0'0 mlcod 0'0 active+remapped+backfilling rops=1 mbc={0={},1={},2={}}] continue_recovery_op: 2:ad551702:::test:head has inconsistent hinfo 2026-02-22T23:30:34.227 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:30:34.229+0000 7fd80d87f640 -1 log_channel(cluster) log [ERR] : 2.0s0 missing primary copy of 2:ad551702:::test:head, will try copies on 1(1),2(2) 2026-02-22T23:30:34.314 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:30:34.315 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:30:34.315 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:30:34.316 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:30:34.403 INFO:teuthology.orchestra.run.trial115.stdout:124554051609 2026-02-22T23:30:34.404 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:34.406 INFO:teuthology.orchestra.run.trial115.stdout:38654705715 2026-02-22T23:30:34.406 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:34.407 INFO:teuthology.orchestra.run.trial115.stdout:98784247850 2026-02-22T23:30:34.408 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:34.410 INFO:teuthology.orchestra.run.trial115.stdout:42949673011 2026-02-22T23:30:34.411 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:34.567 INFO:teuthology.orchestra.run.trial115.stdout:124554051608 2026-02-22T23:30:34.582 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051609 got 124554051608 for osd.3 2026-02-22T23:30:34.603 INFO:teuthology.orchestra.run.trial115.stdout:42949673010 2026-02-22T23:30:34.606 INFO:teuthology.orchestra.run.trial115.stdout:98784247849 2026-02-22T23:30:34.614 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673011 got 42949673010 for osd.0 2026-02-22T23:30:34.617 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247850 got 98784247849 for osd.1 2026-02-22T23:30:34.619 INFO:teuthology.orchestra.run.trial115.stdout:38654705713 2026-02-22T23:30:34.634 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705715 got 38654705713 for osd.2 2026-02-22T23:30:35.583 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:35.615 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:35.617 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:35.635 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:35.750 INFO:teuthology.orchestra.run.trial115.stdout:124554051608 2026-02-22T23:30:35.764 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051609 got 124554051608 for osd.3 2026-02-22T23:30:35.780 INFO:teuthology.orchestra.run.trial115.stdout:98784247849 2026-02-22T23:30:35.790 INFO:teuthology.orchestra.run.trial115.stdout:42949673010 2026-02-22T23:30:35.793 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247850 got 98784247849 for osd.1 2026-02-22T23:30:35.804 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673011 got 42949673010 for osd.0 2026-02-22T23:30:35.825 INFO:teuthology.orchestra.run.trial115.stdout:38654705713 2026-02-22T23:30:35.838 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705715 got 38654705713 for osd.2 2026-02-22T23:30:36.765 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:36.793 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:36.805 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:36.839 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:36.914 INFO:teuthology.orchestra.run.trial115.stdout:124554051609 2026-02-22T23:30:36.929 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051609 got 124554051609 for osd.3 2026-02-22T23:30:36.929 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:36.972 INFO:teuthology.orchestra.run.trial115.stdout:42949673012 2026-02-22T23:30:36.974 INFO:teuthology.orchestra.run.trial115.stdout:98784247850 2026-02-22T23:30:36.987 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673011 got 42949673012 for osd.0 2026-02-22T23:30:36.988 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:36.988 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247850 got 98784247850 for osd.1 2026-02-22T23:30:36.989 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:37.010 INFO:teuthology.orchestra.run.trial115.stdout:38654705715 2026-02-22T23:30:37.022 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705715 got 38654705715 for osd.2 2026-02-22T23:30:37.023 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:37.023 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:30:37.224 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:30:37.225 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:30:37.237 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2026-02-22T23:30:36.157283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22059,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46544,"num_bytes_recovered":381600817,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1184492,"kb_used_data":736248,"kb_used_omap":7,"kb_used_meta":448184,"kb_avail":281931028,"statfs":{"total":289910292480,"available":288697372672,"internally_reserved":0,"allocated":753917952,"data_stored":754020844,"data_compressed":17294,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":7633,"internal_metadata":458940975},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-9437,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9437,"num_bytes_recovered":77308091,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001360"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":161769,"reported_epoch":33,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:30:34.394904+0000","last_change":"2026-02-22T23:29:28.357217+0000","last_active":"2026-02-22T23:30:34.394904+0000","last_peered":"2026-02-22T23:30:34.394904+0000","last_clean":"2026-02-22T23:29:23.243943+0000","last_became_active":"2026-02-22T23:29:28.356529+0000","last_became_peered":"2026-02-22T23:29:28.356529+0000","last_unstale":"2026-02-22T23:30:34.394904+0000","last_undegraded":"2026-02-22T23:30:34.394904+0000","last_fullsized":"2026-02-22T23:30:34.394904+0000","mapping_epoch":32,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22059,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46542,"num_bytes_recovered":381272625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":57,"reported_epoch":33,"state":"active+clean","last_fresh":"2026-02-22T23:29:28.342971+0000","last_change":"2026-02-22T23:29:27.256843+0000","last_active":"2026-02-22T23:29:28.342971+0000","last_peered":"2026-02-22T23:29:28.342971+0000","last_clean":"2026-02-22T23:29:28.342971+0000","last_became_active":"2026-02-22T23:29:27.253029+0000","last_became_peered":"2026-02-22T23:29:27.253029+0000","last_unstale":"2026-02-22T23:29:28.342971+0000","last_undegraded":"2026-02-22T23:29:28.342971+0000","last_fullsized":"2026-02-22T23:29:28.342971+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:36:17.698676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22059,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46542,"num_bytes_recovered":381272625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1035792384,"data_stored":1035825152,"data_compressed":20370,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":655872,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673012,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":430428,"kb_used_data":186892,"kb_used_omap":1,"kb_used_meta":243518,"kb_avail":93941412,"statfs":{"total":96636764160,"available":96196005888,"internally_reserved":0,"allocated":191377408,"data_stored":191511716,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":249362886},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051609,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313596,"kb_used_data":274668,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058244,"statfs":{"total":96636764160,"available":96315641856,"internally_reserved":0,"allocated":281260032,"data_stored":281090724,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440468,"kb_used_data":274688,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931372,"statfs":{"total":96636764160,"available":96185724928,"internally_reserved":0,"allocated":281280512,"data_stored":281418404,"data_compressed":7527,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":98784247850,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192851968,"data_stored":192860160,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:30:37.238 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 161769, 'reported_epoch': 33, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:30:34.394904+0000', 'last_change': '2026-02-22T23:29:28.357217+0000', 'last_active': '2026-02-22T23:30:34.394904+0000', 'last_peered': '2026-02-22T23:30:34.394904+0000', 'last_clean': '2026-02-22T23:29:23.243943+0000', 'last_became_active': '2026-02-22T23:29:28.356529+0000', 'last_became_peered': '2026-02-22T23:29:28.356529+0000', 'last_unstale': '2026-02-22T23:30:34.394904+0000', 'last_undegraded': '2026-02-22T23:30:34.394904+0000', 'last_fullsized': '2026-02-22T23:30:34.394904+0000', 'mapping_epoch': 32, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 30, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 22059, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46542, 'num_bytes_recovered': 381272625, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:30:37.238 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:30:42.239 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:30:42.239 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:30:42.240 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:30:42.240 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:30:42.327 INFO:teuthology.orchestra.run.trial115.stdout:98784247852 2026-02-22T23:30:42.327 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:42.329 INFO:teuthology.orchestra.run.trial115.stdout:42949673014 2026-02-22T23:30:42.329 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:42.331 INFO:teuthology.orchestra.run.trial115.stdout:38654705717 2026-02-22T23:30:42.331 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:42.332 INFO:teuthology.orchestra.run.trial115.stdout:124554051612 2026-02-22T23:30:42.332 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:42.481 INFO:teuthology.orchestra.run.trial115.stdout:98784247851 2026-02-22T23:30:42.481 INFO:teuthology.orchestra.run.trial115.stdout:42949673013 2026-02-22T23:30:42.494 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673014 got 42949673013 for osd.0 2026-02-22T23:30:42.495 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247852 got 98784247851 for osd.1 2026-02-22T23:30:42.522 INFO:teuthology.orchestra.run.trial115.stdout:38654705716 2026-02-22T23:30:42.532 INFO:teuthology.orchestra.run.trial115.stdout:124554051611 2026-02-22T23:30:42.533 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705717 got 38654705716 for osd.2 2026-02-22T23:30:42.547 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051612 got 124554051611 for osd.3 2026-02-22T23:30:43.495 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:43.496 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:43.534 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:43.548 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:43.648 INFO:teuthology.orchestra.run.trial115.stdout:98784247851 2026-02-22T23:30:43.661 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247852 got 98784247851 for osd.1 2026-02-22T23:30:43.695 INFO:teuthology.orchestra.run.trial115.stdout:42949673013 2026-02-22T23:30:43.709 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673014 got 42949673013 for osd.0 2026-02-22T23:30:43.738 INFO:teuthology.orchestra.run.trial115.stdout:38654705716 2026-02-22T23:30:43.746 INFO:teuthology.orchestra.run.trial115.stdout:124554051611 2026-02-22T23:30:43.751 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705717 got 38654705716 for osd.2 2026-02-22T23:30:43.759 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051612 got 124554051611 for osd.3 2026-02-22T23:30:44.663 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:44.710 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:44.752 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:44.760 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:44.818 INFO:teuthology.orchestra.run.trial115.stdout:98784247853 2026-02-22T23:30:44.832 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247852 got 98784247853 for osd.1 2026-02-22T23:30:44.832 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:44.877 INFO:teuthology.orchestra.run.trial115.stdout:42949673014 2026-02-22T23:30:44.891 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673014 got 42949673014 for osd.0 2026-02-22T23:30:44.891 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:44.922 INFO:teuthology.orchestra.run.trial115.stdout:38654705717 2026-02-22T23:30:44.937 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705717 got 38654705717 for osd.2 2026-02-22T23:30:44.937 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:44.964 INFO:teuthology.orchestra.run.trial115.stdout:124554051612 2026-02-22T23:30:44.976 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051612 got 124554051612 for osd.3 2026-02-22T23:30:44.976 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:44.977 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:30:45.168 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:30:45.169 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:30:45.179 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2026-02-22T23:30:44.158247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16580,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52023,"num_bytes_recovered":426476617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1244112,"kb_used_data":755612,"kb_used_omap":7,"kb_used_meta":488440,"kb_avail":281871408,"statfs":{"total":289910292480,"available":288636321792,"internally_reserved":0,"allocated":773746688,"data_stored":773849580,"data_compressed":17294,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":7633,"internal_metadata":500163119},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-7755,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7755,"num_bytes_recovered":63520979,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001433"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":172727,"reported_epoch":33,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:30:42.324575+0000","last_change":"2026-02-22T23:29:28.357217+0000","last_active":"2026-02-22T23:30:42.324575+0000","last_peered":"2026-02-22T23:30:42.324575+0000","last_clean":"2026-02-22T23:29:23.243943+0000","last_became_active":"2026-02-22T23:29:28.356529+0000","last_became_peered":"2026-02-22T23:29:28.356529+0000","last_unstale":"2026-02-22T23:30:42.324575+0000","last_undegraded":"2026-02-22T23:30:42.324575+0000","last_fullsized":"2026-02-22T23:30:42.324575+0000","mapping_epoch":32,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16580,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52021,"num_bytes_recovered":426148425,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":57,"reported_epoch":33,"state":"active+clean","last_fresh":"2026-02-22T23:29:28.342971+0000","last_change":"2026-02-22T23:29:27.256843+0000","last_active":"2026-02-22T23:29:28.342971+0000","last_peered":"2026-02-22T23:29:28.342971+0000","last_clean":"2026-02-22T23:29:28.342971+0000","last_became_active":"2026-02-22T23:29:27.253029+0000","last_became_peered":"2026-02-22T23:29:27.253029+0000","last_unstale":"2026-02-22T23:29:28.342971+0000","last_undegraded":"2026-02-22T23:29:28.342971+0000","last_fullsized":"2026-02-22T23:29:28.342971+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:36:17.698676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16580,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52021,"num_bytes_recovered":426148425,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1056006144,"data_stored":1056038912,"data_compressed":20370,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":655872,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673014,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490048,"kb_used_data":206256,"kb_used_omap":1,"kb_used_meta":283774,"kb_avail":93881792,"statfs":{"total":96636764160,"available":96134955008,"internally_reserved":0,"allocated":211206144,"data_stored":211340452,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":290585030},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051612,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313596,"kb_used_data":274668,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058244,"statfs":{"total":96636764160,"available":96315641856,"internally_reserved":0,"allocated":281260032,"data_stored":281090724,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440468,"kb_used_data":274688,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931372,"statfs":{"total":96636764160,"available":96185724928,"internally_reserved":0,"allocated":281280512,"data_stored":281418404,"data_compressed":7527,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":98784247853,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":213065728,"data_stored":213073920,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:30:45.180 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 172727, 'reported_epoch': 33, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:30:42.324575+0000', 'last_change': '2026-02-22T23:29:28.357217+0000', 'last_active': '2026-02-22T23:30:42.324575+0000', 'last_peered': '2026-02-22T23:30:42.324575+0000', 'last_clean': '2026-02-22T23:29:23.243943+0000', 'last_became_active': '2026-02-22T23:29:28.356529+0000', 'last_became_peered': '2026-02-22T23:29:28.356529+0000', 'last_unstale': '2026-02-22T23:30:42.324575+0000', 'last_undegraded': '2026-02-22T23:30:42.324575+0000', 'last_fullsized': '2026-02-22T23:30:42.324575+0000', 'mapping_epoch': 32, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 30, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 16580, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 52021, 'num_bytes_recovered': 426148425, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:30:45.181 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:30:50.182 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:30:50.183 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:30:50.183 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:30:50.184 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:30:50.271 INFO:teuthology.orchestra.run.trial115.stdout:38654705720 2026-02-22T23:30:50.272 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:50.273 INFO:teuthology.orchestra.run.trial115.stdout:124554051614 2026-02-22T23:30:50.273 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:50.275 INFO:teuthology.orchestra.run.trial115.stdout:98784247855 2026-02-22T23:30:50.276 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:50.278 INFO:teuthology.orchestra.run.trial115.stdout:42949673017 2026-02-22T23:30:50.278 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:50.426 INFO:teuthology.orchestra.run.trial115.stdout:124554051613 2026-02-22T23:30:50.438 INFO:teuthology.orchestra.run.trial115.stdout:38654705719 2026-02-22T23:30:50.439 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051614 got 124554051613 for osd.3 2026-02-22T23:30:50.452 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705720 got 38654705719 for osd.2 2026-02-22T23:30:50.474 INFO:teuthology.orchestra.run.trial115.stdout:42949673015 2026-02-22T23:30:50.477 INFO:teuthology.orchestra.run.trial115.stdout:98784247854 2026-02-22T23:30:50.488 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673017 got 42949673015 for osd.0 2026-02-22T23:30:50.491 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247855 got 98784247854 for osd.1 2026-02-22T23:30:51.440 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:51.452 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:51.488 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:51.492 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:51.594 INFO:teuthology.orchestra.run.trial115.stdout:124554051613 2026-02-22T23:30:51.608 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051614 got 124554051613 for osd.3 2026-02-22T23:30:51.643 INFO:teuthology.orchestra.run.trial115.stdout:38654705719 2026-02-22T23:30:51.657 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705720 got 38654705719 for osd.2 2026-02-22T23:30:51.678 INFO:teuthology.orchestra.run.trial115.stdout:98784247854 2026-02-22T23:30:51.679 INFO:teuthology.orchestra.run.trial115.stdout:42949673015 2026-02-22T23:30:51.692 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247855 got 98784247854 for osd.1 2026-02-22T23:30:51.693 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673017 got 42949673015 for osd.0 2026-02-22T23:30:52.609 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:52.658 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:52.693 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:52.694 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:52.766 INFO:teuthology.orchestra.run.trial115.stdout:124554051615 2026-02-22T23:30:52.780 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051614 got 124554051615 for osd.3 2026-02-22T23:30:52.780 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:52.809 INFO:teuthology.orchestra.run.trial115.stdout:38654705720 2026-02-22T23:30:52.824 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705720 got 38654705720 for osd.2 2026-02-22T23:30:52.824 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:52.859 INFO:teuthology.orchestra.run.trial115.stdout:98784247855 2026-02-22T23:30:52.868 INFO:teuthology.orchestra.run.trial115.stdout:42949673017 2026-02-22T23:30:52.873 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247855 got 98784247855 for osd.1 2026-02-22T23:30:52.873 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:52.882 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673017 got 42949673017 for osd.0 2026-02-22T23:30:52.882 DEBUG:teuthology.parallel:result is None 2026-02-22T23:30:52.883 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:30:53.074 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:30:53.074 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:30:53.085 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2026-02-22T23:30:52.159203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10233,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58370,"num_bytes_recovered":478471241,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1262348,"kb_used_data":778008,"kb_used_omap":7,"kb_used_meta":484280,"kb_avail":281853172,"statfs":{"total":289910292480,"available":288617648128,"internally_reserved":0,"allocated":796680192,"data_stored":796783084,"data_compressed":17294,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":7633,"internal_metadata":495903279},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-10551,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10551,"num_bytes_recovered":86425624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001416"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":185421,"reported_epoch":33,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:30:51.241319+0000","last_change":"2026-02-22T23:29:28.357217+0000","last_active":"2026-02-22T23:30:51.241319+0000","last_peered":"2026-02-22T23:30:51.241319+0000","last_clean":"2026-02-22T23:29:23.243943+0000","last_became_active":"2026-02-22T23:29:28.356529+0000","last_became_peered":"2026-02-22T23:29:28.356529+0000","last_unstale":"2026-02-22T23:30:51.241319+0000","last_undegraded":"2026-02-22T23:30:51.241319+0000","last_fullsized":"2026-02-22T23:30:51.241319+0000","mapping_epoch":32,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10233,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58368,"num_bytes_recovered":478143049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":57,"reported_epoch":33,"state":"active+clean","last_fresh":"2026-02-22T23:29:28.342971+0000","last_change":"2026-02-22T23:29:27.256843+0000","last_active":"2026-02-22T23:29:28.342971+0000","last_peered":"2026-02-22T23:29:28.342971+0000","last_clean":"2026-02-22T23:29:28.342971+0000","last_became_active":"2026-02-22T23:29:27.253029+0000","last_became_peered":"2026-02-22T23:29:27.253029+0000","last_unstale":"2026-02-22T23:29:28.342971+0000","last_undegraded":"2026-02-22T23:29:28.342971+0000","last_fullsized":"2026-02-22T23:29:28.342971+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:36:17.698676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10233,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58368,"num_bytes_recovered":478143049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1079111680,"data_stored":1079144448,"data_compressed":20370,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":655872,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673017,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":508284,"kb_used_data":228652,"kb_used_omap":1,"kb_used_meta":279614,"kb_avail":93863556,"statfs":{"total":96636764160,"available":96116281344,"internally_reserved":0,"allocated":234139648,"data_stored":234273956,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":286325190},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051615,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313596,"kb_used_data":274668,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058244,"statfs":{"total":96636764160,"available":96315641856,"internally_reserved":0,"allocated":281260032,"data_stored":281090724,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440468,"kb_used_data":274688,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931372,"statfs":{"total":96636764160,"available":96185724928,"internally_reserved":0,"allocated":281280512,"data_stored":281418404,"data_compressed":7527,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":98784247855,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":236171264,"data_stored":236179456,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:30:53.087 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 185421, 'reported_epoch': 33, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:30:51.241319+0000', 'last_change': '2026-02-22T23:29:28.357217+0000', 'last_active': '2026-02-22T23:30:51.241319+0000', 'last_peered': '2026-02-22T23:30:51.241319+0000', 'last_clean': '2026-02-22T23:29:23.243943+0000', 'last_became_active': '2026-02-22T23:29:28.356529+0000', 'last_became_peered': '2026-02-22T23:29:28.356529+0000', 'last_unstale': '2026-02-22T23:30:51.241319+0000', 'last_undegraded': '2026-02-22T23:30:51.241319+0000', 'last_fullsized': '2026-02-22T23:30:51.241319+0000', 'mapping_epoch': 32, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 30, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 10233, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 58368, 'num_bytes_recovered': 478143049, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:30:53.087 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:30:58.088 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:30:58.088 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:30:58.089 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:30:58.089 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:30:58.177 INFO:teuthology.orchestra.run.trial115.stdout:38654705722 2026-02-22T23:30:58.177 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:58.179 INFO:teuthology.orchestra.run.trial115.stdout:98784247858 2026-02-22T23:30:58.180 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:58.181 INFO:teuthology.orchestra.run.trial115.stdout:124554051617 2026-02-22T23:30:58.181 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:58.184 INFO:teuthology.orchestra.run.trial115.stdout:42949673019 2026-02-22T23:30:58.184 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:58.331 INFO:teuthology.orchestra.run.trial115.stdout:38654705721 2026-02-22T23:30:58.344 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705722 got 38654705721 for osd.2 2026-02-22T23:30:58.389 INFO:teuthology.orchestra.run.trial115.stdout:124554051616 2026-02-22T23:30:58.393 INFO:teuthology.orchestra.run.trial115.stdout:98784247856 2026-02-22T23:30:58.399 INFO:teuthology.orchestra.run.trial115.stdout:42949673018 2026-02-22T23:30:58.405 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051617 got 124554051616 for osd.3 2026-02-22T23:30:58.409 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247858 got 98784247856 for osd.1 2026-02-22T23:30:58.415 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673019 got 42949673018 for osd.0 2026-02-22T23:30:59.346 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:30:59.405 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:30:59.410 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:30:59.416 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:30:59.490 INFO:teuthology.orchestra.run.trial115.stdout:38654705721 2026-02-22T23:30:59.504 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705722 got 38654705721 for osd.2 2026-02-22T23:30:59.566 INFO:teuthology.orchestra.run.trial115.stdout:124554051616 2026-02-22T23:30:59.580 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051617 got 124554051616 for osd.3 2026-02-22T23:30:59.601 INFO:teuthology.orchestra.run.trial115.stdout:42949673018 2026-02-22T23:30:59.614 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673019 got 42949673018 for osd.0 2026-02-22T23:30:59.620 INFO:teuthology.orchestra.run.trial115.stdout:98784247856 2026-02-22T23:30:59.634 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247858 got 98784247856 for osd.1 2026-02-22T23:31:00.505 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:31:00.581 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:31:00.615 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:31:00.635 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:31:00.655 INFO:teuthology.orchestra.run.trial115.stdout:38654705723 2026-02-22T23:31:00.669 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705722 got 38654705723 for osd.2 2026-02-22T23:31:00.669 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:00.729 INFO:teuthology.orchestra.run.trial115.stdout:124554051617 2026-02-22T23:31:00.743 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051617 got 124554051617 for osd.3 2026-02-22T23:31:00.743 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:00.780 INFO:teuthology.orchestra.run.trial115.stdout:42949673019 2026-02-22T23:31:00.793 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673019 got 42949673019 for osd.0 2026-02-22T23:31:00.794 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:00.816 INFO:teuthology.orchestra.run.trial115.stdout:98784247858 2026-02-22T23:31:00.829 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247858 got 98784247858 for osd.1 2026-02-22T23:31:00.829 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:00.830 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:31:01.025 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:01.025 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:31:01.038 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2026-02-22T23:31:00.160184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5344,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63259,"num_bytes_recovered":518521929,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1338248,"kb_used_data":801172,"kb_used_omap":7,"kb_used_meta":537016,"kb_avail":281777272,"statfs":{"total":289910292480,"available":288539926528,"internally_reserved":0,"allocated":820400128,"data_stored":820503020,"data_compressed":17294,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":7633,"internal_metadata":549904943},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-8444,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8444,"num_bytes_recovered":69173248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001434"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":195199,"reported_epoch":33,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:30:58.171355+0000","last_change":"2026-02-22T23:29:28.357217+0000","last_active":"2026-02-22T23:30:58.171355+0000","last_peered":"2026-02-22T23:30:58.171355+0000","last_clean":"2026-02-22T23:29:23.243943+0000","last_became_active":"2026-02-22T23:29:28.356529+0000","last_became_peered":"2026-02-22T23:29:28.356529+0000","last_unstale":"2026-02-22T23:30:58.171355+0000","last_undegraded":"2026-02-22T23:30:58.171355+0000","last_fullsized":"2026-02-22T23:30:58.171355+0000","mapping_epoch":32,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5344,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63257,"num_bytes_recovered":518193737,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":57,"reported_epoch":33,"state":"active+clean","last_fresh":"2026-02-22T23:29:28.342971+0000","last_change":"2026-02-22T23:29:27.256843+0000","last_active":"2026-02-22T23:29:28.342971+0000","last_peered":"2026-02-22T23:29:28.342971+0000","last_clean":"2026-02-22T23:29:28.342971+0000","last_became_active":"2026-02-22T23:29:27.253029+0000","last_became_peered":"2026-02-22T23:29:27.253029+0000","last_unstale":"2026-02-22T23:29:28.342971+0000","last_undegraded":"2026-02-22T23:29:28.342971+0000","last_fullsized":"2026-02-22T23:29:28.342971+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:36:17.698676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5344,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63257,"num_bytes_recovered":518193737,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1102045184,"data_stored":1102077952,"data_compressed":20370,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":655872,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673019,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584184,"kb_used_data":251816,"kb_used_omap":1,"kb_used_meta":332350,"kb_avail":93787656,"statfs":{"total":96636764160,"available":96038559744,"internally_reserved":0,"allocated":257859584,"data_stored":257993892,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":340326854},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051617,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313596,"kb_used_data":274668,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058244,"statfs":{"total":96636764160,"available":96315641856,"internally_reserved":0,"allocated":281260032,"data_stored":281090724,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440468,"kb_used_data":274688,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931372,"statfs":{"total":96636764160,"available":96185724928,"internally_reserved":0,"allocated":281280512,"data_stored":281418404,"data_compressed":7527,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":98784247858,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":259104768,"data_stored":259112960,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:31:01.039 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 195199, 'reported_epoch': 33, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:30:58.171355+0000', 'last_change': '2026-02-22T23:29:28.357217+0000', 'last_active': '2026-02-22T23:30:58.171355+0000', 'last_peered': '2026-02-22T23:30:58.171355+0000', 'last_clean': '2026-02-22T23:29:23.243943+0000', 'last_became_active': '2026-02-22T23:29:28.356529+0000', 'last_became_peered': '2026-02-22T23:29:28.356529+0000', 'last_unstale': '2026-02-22T23:30:58.171355+0000', 'last_undegraded': '2026-02-22T23:30:58.171355+0000', 'last_fullsized': '2026-02-22T23:30:58.171355+0000', 'mapping_epoch': 32, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 30, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5344, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 63257, 'num_bytes_recovered': 518193737, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:31:01.039 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:31:06.040 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:31:06.041 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:31:06.042 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:31:06.042 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:31:06.129 INFO:teuthology.orchestra.run.trial115.stdout:38654705725 2026-02-22T23:31:06.129 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:31:06.130 INFO:teuthology.orchestra.run.trial115.stdout:42949673022 2026-02-22T23:31:06.131 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:31:06.132 INFO:teuthology.orchestra.run.trial115.stdout:98784247860 2026-02-22T23:31:06.132 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:31:06.138 INFO:teuthology.orchestra.run.trial115.stdout:124554051619 2026-02-22T23:31:06.139 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:31:06.279 INFO:teuthology.orchestra.run.trial115.stdout:38654705724 2026-02-22T23:31:06.292 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705725 got 38654705724 for osd.2 2026-02-22T23:31:06.325 INFO:teuthology.orchestra.run.trial115.stdout:42949673020 2026-02-22T23:31:06.329 INFO:teuthology.orchestra.run.trial115.stdout:98784247859 2026-02-22T23:31:06.331 INFO:teuthology.orchestra.run.trial115.stdout:124554051618 2026-02-22T23:31:06.337 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673022 got 42949673020 for osd.0 2026-02-22T23:31:06.341 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247860 got 98784247859 for osd.1 2026-02-22T23:31:06.343 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051619 got 124554051618 for osd.3 2026-02-22T23:31:07.292 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:31:07.337 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:31:07.342 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:31:07.344 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:31:07.451 INFO:teuthology.orchestra.run.trial115.stdout:38654705725 2026-02-22T23:31:07.464 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705725 got 38654705725 for osd.2 2026-02-22T23:31:07.465 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:07.480 INFO:teuthology.orchestra.run.trial115.stdout:42949673022 2026-02-22T23:31:07.492 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673022 got 42949673022 for osd.0 2026-02-22T23:31:07.492 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:07.530 INFO:teuthology.orchestra.run.trial115.stdout:124554051619 2026-02-22T23:31:07.535 INFO:teuthology.orchestra.run.trial115.stdout:98784247860 2026-02-22T23:31:07.541 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051619 got 124554051619 for osd.3 2026-02-22T23:31:07.541 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:07.547 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247860 got 98784247860 for osd.1 2026-02-22T23:31:07.547 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:07.548 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:31:07.736 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:07.737 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:31:07.746 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2026-02-22T23:31:06.899429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68602,"num_bytes_recovered":562291785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1392296,"kb_used_data":823280,"kb_used_omap":7,"kb_used_meta":568952,"kb_avail":281723224,"statfs":{"total":289910292480,"available":288484581376,"internally_reserved":0,"allocated":843038720,"data_stored":843136880,"data_compressed":17294,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":7633,"internal_metadata":582607407},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6686,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6685,"num_bytes_recovered":54763520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.739511"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":205889,"reported_epoch":33,"state":"peering","last_fresh":"2026-02-22T23:31:05.842930+0000","last_change":"2026-02-22T23:31:05.895284+0000","last_active":"2026-02-22T23:31:05.895284+0000","last_peered":"2026-02-22T23:31:05.842930+0000","last_clean":"2026-02-22T23:31:05.842930+0000","last_became_active":"2026-02-22T23:29:28.356529+0000","last_became_peered":"2026-02-22T23:29:28.356529+0000","last_unstale":"2026-02-22T23:31:05.842930+0000","last_undegraded":"2026-02-22T23:31:05.842930+0000","last_fullsized":"2026-02-22T23:31:05.842930+0000","mapping_epoch":34,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68600,"num_bytes_recovered":561963593,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-22T23:31:05.896761+0000","last_change":"2026-02-22T23:29:27.256843+0000","last_active":"2026-02-22T23:31:05.896761+0000","last_peered":"2026-02-22T23:31:05.896761+0000","last_clean":"2026-02-22T23:31:05.896761+0000","last_became_active":"2026-02-22T23:29:27.253029+0000","last_became_peered":"2026-02-22T23:29:27.253029+0000","last_unstale":"2026-02-22T23:31:05.896761+0000","last_undegraded":"2026-02-22T23:31:05.896761+0000","last_fullsized":"2026-02-22T23:31:05.896761+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:36:17.698676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68600,"num_bytes_recovered":561963593,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123917824,"data_stored":1123950592,"data_compressed":20370,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":655872,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673022,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":638220,"kb_used_data":273916,"kb_used_omap":1,"kb_used_meta":364286,"kb_avail":93733620,"statfs":{"total":96636764160,"available":95983226880,"internally_reserved":0,"allocated":280489984,"data_stored":280624292,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":373029318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051620,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313608,"kb_used_data":274676,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058232,"statfs":{"total":96636764160,"available":96315629568,"internally_reserved":0,"allocated":281268224,"data_stored":281094184,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440468,"kb_used_data":274688,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931372,"statfs":{"total":96636764160,"available":96185724928,"internally_reserved":0,"allocated":281280512,"data_stored":281418404,"data_compressed":7527,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":98784247860,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:31:07.747 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 205889, 'reported_epoch': 33, 'state': 'peering', 'last_fresh': '2026-02-22T23:31:05.842930+0000', 'last_change': '2026-02-22T23:31:05.895284+0000', 'last_active': '2026-02-22T23:31:05.895284+0000', 'last_peered': '2026-02-22T23:31:05.842930+0000', 'last_clean': '2026-02-22T23:31:05.842930+0000', 'last_became_active': '2026-02-22T23:29:28.356529+0000', 'last_became_peered': '2026-02-22T23:29:28.356529+0000', 'last_unstale': '2026-02-22T23:31:05.842930+0000', 'last_undegraded': '2026-02-22T23:31:05.842930+0000', 'last_fullsized': '2026-02-22T23:31:05.842930+0000', 'mapping_epoch': 34, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 68600, 'num_bytes_recovered': 561963593, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:31:07.748 DEBUG:tasks.ec_inconsistent_hinfo:not active 2026-02-22T23:31:12.749 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:31:12.749 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:31:12.750 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:31:12.750 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:31:12.836 INFO:teuthology.orchestra.run.trial115.stdout:38654705727 2026-02-22T23:31:12.836 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:31:12.840 INFO:teuthology.orchestra.run.trial115.stdout:98784247862 2026-02-22T23:31:12.841 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:31:12.843 INFO:teuthology.orchestra.run.trial115.stdout:124554051622 2026-02-22T23:31:12.843 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:31:12.849 INFO:teuthology.orchestra.run.trial115.stdout:42949673024 2026-02-22T23:31:12.850 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:31:12.986 INFO:teuthology.orchestra.run.trial115.stdout:38654705726 2026-02-22T23:31:12.991 INFO:teuthology.orchestra.run.trial115.stdout:98784247861 2026-02-22T23:31:12.999 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705727 got 38654705726 for osd.2 2026-02-22T23:31:13.003 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247862 got 98784247861 for osd.1 2026-02-22T23:31:13.033 INFO:teuthology.orchestra.run.trial115.stdout:124554051621 2026-02-22T23:31:13.045 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051622 got 124554051621 for osd.3 2026-02-22T23:31:13.046 INFO:teuthology.orchestra.run.trial115.stdout:42949673023 2026-02-22T23:31:13.058 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673024 got 42949673023 for osd.0 2026-02-22T23:31:14.000 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:31:14.004 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:31:14.046 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:31:14.059 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:31:14.158 INFO:teuthology.orchestra.run.trial115.stdout:38654705726 2026-02-22T23:31:14.171 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705727 got 38654705726 for osd.2 2026-02-22T23:31:14.188 INFO:teuthology.orchestra.run.trial115.stdout:98784247861 2026-02-22T23:31:14.202 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247862 got 98784247861 for osd.1 2026-02-22T23:31:14.229 INFO:teuthology.orchestra.run.trial115.stdout:124554051621 2026-02-22T23:31:14.235 INFO:teuthology.orchestra.run.trial115.stdout:42949673023 2026-02-22T23:31:14.241 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051622 got 124554051621 for osd.3 2026-02-22T23:31:14.247 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673024 got 42949673023 for osd.0 2026-02-22T23:31:15.171 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:31:15.202 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:31:15.242 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:31:15.248 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:31:15.318 INFO:teuthology.orchestra.run.trial115.stdout:38654705728 2026-02-22T23:31:15.330 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705727 got 38654705728 for osd.2 2026-02-22T23:31:15.330 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:15.362 INFO:teuthology.orchestra.run.trial115.stdout:98784247863 2026-02-22T23:31:15.375 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247862 got 98784247863 for osd.1 2026-02-22T23:31:15.375 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:15.404 INFO:teuthology.orchestra.run.trial115.stdout:42949673024 2026-02-22T23:31:15.409 INFO:teuthology.orchestra.run.trial115.stdout:124554051622 2026-02-22T23:31:15.416 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673024 got 42949673024 for osd.0 2026-02-22T23:31:15.416 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:15.423 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051622 got 124554051622 for osd.3 2026-02-22T23:31:15.423 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:15.423 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:31:15.610 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:15.610 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:31:15.620 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2026-02-22T23:31:14.161945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68602,"num_bytes_recovered":562291785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1393120,"kb_used_data":824092,"kb_used_omap":7,"kb_used_meta":568952,"kb_avail":281722400,"statfs":{"total":289910292480,"available":288483737600,"internally_reserved":0,"allocated":843870208,"data_stored":843944645,"data_compressed":17294,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":7633,"internal_metadata":582607407},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3218,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3217,"num_bytes_recovered":26353664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.267527"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":205898,"reported_epoch":35,"state":"active+clean","last_fresh":"2026-02-22T23:31:06.902794+0000","last_change":"2026-02-22T23:31:06.902794+0000","last_active":"2026-02-22T23:31:06.902794+0000","last_peered":"2026-02-22T23:31:06.902794+0000","last_clean":"2026-02-22T23:31:06.902794+0000","last_became_active":"2026-02-22T23:31:06.902580+0000","last_became_peered":"2026-02-22T23:31:06.902580+0000","last_unstale":"2026-02-22T23:31:06.902794+0000","last_undegraded":"2026-02-22T23:31:06.902794+0000","last_fullsized":"2026-02-22T23:31:06.902794+0000","mapping_epoch":34,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:51:24.820525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68600,"num_bytes_recovered":561963593,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":61,"reported_epoch":35,"state":"active+clean","last_fresh":"2026-02-22T23:31:06.901992+0000","last_change":"2026-02-22T23:29:27.256843+0000","last_active":"2026-02-22T23:31:06.901992+0000","last_peered":"2026-02-22T23:31:06.901992+0000","last_clean":"2026-02-22T23:31:06.901992+0000","last_became_active":"2026-02-22T23:29:27.253029+0000","last_became_peered":"2026-02-22T23:29:27.253029+0000","last_unstale":"2026-02-22T23:31:06.901992+0000","last_undegraded":"2026-02-22T23:31:06.901992+0000","last_fullsized":"2026-02-22T23:31:06.901992+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:36:17.698676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68600,"num_bytes_recovered":561963593,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123061760,"data_stored":1123094528,"data_compressed":20370,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":655872,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673024,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":639016,"kb_used_data":274704,"kb_used_omap":1,"kb_used_meta":364286,"kb_avail":93732824,"statfs":{"total":96636764160,"available":95982411776,"internally_reserved":0,"allocated":281296896,"data_stored":281421719,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":373029318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051622,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313620,"kb_used_data":274684,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058220,"statfs":{"total":96636764160,"available":96315617280,"internally_reserved":0,"allocated":281276416,"data_stored":281097623,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440484,"kb_used_data":274704,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931356,"statfs":{"total":96636764160,"available":96185708544,"internally_reserved":0,"allocated":281296896,"data_stored":281425303,"data_compressed":7527,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":98784247863,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280121344,"data_stored":280129536,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:31:15.621 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 205898, 'reported_epoch': 35, 'state': 'active+clean', 'last_fresh': '2026-02-22T23:31:06.902794+0000', 'last_change': '2026-02-22T23:31:06.902794+0000', 'last_active': '2026-02-22T23:31:06.902794+0000', 'last_peered': '2026-02-22T23:31:06.902794+0000', 'last_clean': '2026-02-22T23:31:06.902794+0000', 'last_became_active': '2026-02-22T23:31:06.902580+0000', 'last_became_peered': '2026-02-22T23:31:06.902580+0000', 'last_unstale': '2026-02-22T23:31:06.902794+0000', 'last_undegraded': '2026-02-22T23:31:06.902794+0000', 'last_fullsized': '2026-02-22T23:31:06.902794+0000', 'mapping_epoch': 34, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 35, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-24T07:51:24.820525+0000', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 68600, 'num_bytes_recovered': 561963593, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:31:15.622 DEBUG:tasks.ec_inconsistent_hinfo:backfilled! 2026-02-22T23:31:15.622 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for clean 2026-02-22T23:31:15.622 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:31:15.764 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:15.765 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:31:15.777 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2026-02-22T23:31:14.161945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68602,"num_bytes_recovered":562291785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1393120,"kb_used_data":824092,"kb_used_omap":7,"kb_used_meta":568952,"kb_avail":281722400,"statfs":{"total":289910292480,"available":288483737600,"internally_reserved":0,"allocated":843870208,"data_stored":843944645,"data_compressed":17294,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":7633,"internal_metadata":582607407},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3218,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3217,"num_bytes_recovered":26353664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.267527"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":205898,"reported_epoch":35,"state":"active+clean","last_fresh":"2026-02-22T23:31:06.902794+0000","last_change":"2026-02-22T23:31:06.902794+0000","last_active":"2026-02-22T23:31:06.902794+0000","last_peered":"2026-02-22T23:31:06.902794+0000","last_clean":"2026-02-22T23:31:06.902794+0000","last_became_active":"2026-02-22T23:31:06.902580+0000","last_became_peered":"2026-02-22T23:31:06.902580+0000","last_unstale":"2026-02-22T23:31:06.902794+0000","last_undegraded":"2026-02-22T23:31:06.902794+0000","last_fullsized":"2026-02-22T23:31:06.902794+0000","mapping_epoch":34,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:51:24.820525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68600,"num_bytes_recovered":561963593,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":61,"reported_epoch":35,"state":"active+clean","last_fresh":"2026-02-22T23:31:06.901992+0000","last_change":"2026-02-22T23:29:27.256843+0000","last_active":"2026-02-22T23:31:06.901992+0000","last_peered":"2026-02-22T23:31:06.901992+0000","last_clean":"2026-02-22T23:31:06.901992+0000","last_became_active":"2026-02-22T23:29:27.253029+0000","last_became_peered":"2026-02-22T23:29:27.253029+0000","last_unstale":"2026-02-22T23:31:06.901992+0000","last_undegraded":"2026-02-22T23:31:06.901992+0000","last_fullsized":"2026-02-22T23:31:06.901992+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:36:17.698676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68600,"num_bytes_recovered":561963593,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123061760,"data_stored":1123094528,"data_compressed":20370,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":655872,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673024,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":639016,"kb_used_data":274704,"kb_used_omap":1,"kb_used_meta":364286,"kb_avail":93732824,"statfs":{"total":96636764160,"available":95982411776,"internally_reserved":0,"allocated":281296896,"data_stored":281421719,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":373029318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051622,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313620,"kb_used_data":274684,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058220,"statfs":{"total":96636764160,"available":96315617280,"internally_reserved":0,"allocated":281276416,"data_stored":281097623,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440484,"kb_used_data":274704,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931356,"statfs":{"total":96636764160,"available":96185708544,"internally_reserved":0,"allocated":281296896,"data_stored":281425303,"data_compressed":7527,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":98784247863,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280121344,"data_stored":280129536,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:31:15.778 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:31:15.919 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:15.919 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:31:15.929 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2026-02-22T23:31:14.161945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68602,"num_bytes_recovered":562291785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1393120,"kb_used_data":824092,"kb_used_omap":7,"kb_used_meta":568952,"kb_avail":281722400,"statfs":{"total":289910292480,"available":288483737600,"internally_reserved":0,"allocated":843870208,"data_stored":843944645,"data_compressed":17294,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":7633,"internal_metadata":582607407},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3218,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3217,"num_bytes_recovered":26353664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.267527"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":205898,"reported_epoch":35,"state":"active+clean","last_fresh":"2026-02-22T23:31:06.902794+0000","last_change":"2026-02-22T23:31:06.902794+0000","last_active":"2026-02-22T23:31:06.902794+0000","last_peered":"2026-02-22T23:31:06.902794+0000","last_clean":"2026-02-22T23:31:06.902794+0000","last_became_active":"2026-02-22T23:31:06.902580+0000","last_became_peered":"2026-02-22T23:31:06.902580+0000","last_unstale":"2026-02-22T23:31:06.902794+0000","last_undegraded":"2026-02-22T23:31:06.902794+0000","last_fullsized":"2026-02-22T23:31:06.902794+0000","mapping_epoch":34,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:51:24.820525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68600,"num_bytes_recovered":561963593,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":61,"reported_epoch":35,"state":"active+clean","last_fresh":"2026-02-22T23:31:06.901992+0000","last_change":"2026-02-22T23:29:27.256843+0000","last_active":"2026-02-22T23:31:06.901992+0000","last_peered":"2026-02-22T23:31:06.901992+0000","last_clean":"2026-02-22T23:31:06.901992+0000","last_became_active":"2026-02-22T23:29:27.253029+0000","last_became_peered":"2026-02-22T23:29:27.253029+0000","last_unstale":"2026-02-22T23:31:06.901992+0000","last_undegraded":"2026-02-22T23:31:06.901992+0000","last_fullsized":"2026-02-22T23:31:06.901992+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:36:17.698676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68600,"num_bytes_recovered":561963593,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123061760,"data_stored":1123094528,"data_compressed":20370,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":655872,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673024,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":639016,"kb_used_data":274704,"kb_used_omap":1,"kb_used_meta":364286,"kb_avail":93732824,"statfs":{"total":96636764160,"available":95982411776,"internally_reserved":0,"allocated":281296896,"data_stored":281421719,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":373029318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051622,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313620,"kb_used_data":274684,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058220,"statfs":{"total":96636764160,"available":96315617280,"internally_reserved":0,"allocated":281276416,"data_stored":281097623,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440484,"kb_used_data":274704,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931356,"statfs":{"total":96636764160,"available":96185708544,"internally_reserved":0,"allocated":281296896,"data_stored":281425303,"data_compressed":7527,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":98784247863,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280121344,"data_stored":280129536,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:31:15.930 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:clean! 2026-02-22T23:31:15.930 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:31:15.931 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:31:15.931 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:31:15.932 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:31:16.014 INFO:teuthology.orchestra.run.trial115.stdout:124554051623 2026-02-22T23:31:16.015 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:31:16.017 INFO:teuthology.orchestra.run.trial115.stdout:98784247864 2026-02-22T23:31:16.018 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:31:16.019 INFO:teuthology.orchestra.run.trial115.stdout:42949673026 2026-02-22T23:31:16.019 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:31:16.024 INFO:teuthology.orchestra.run.trial115.stdout:38654705729 2026-02-22T23:31:16.024 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:31:16.174 INFO:teuthology.orchestra.run.trial115.stdout:124554051622 2026-02-22T23:31:16.186 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051623 got 124554051622 for osd.3 2026-02-22T23:31:16.208 INFO:teuthology.orchestra.run.trial115.stdout:98784247863 2026-02-22T23:31:16.210 INFO:teuthology.orchestra.run.trial115.stdout:42949673024 2026-02-22T23:31:16.215 INFO:teuthology.orchestra.run.trial115.stdout:38654705728 2026-02-22T23:31:16.220 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247864 got 98784247863 for osd.1 2026-02-22T23:31:16.221 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673026 got 42949673024 for osd.0 2026-02-22T23:31:16.227 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705729 got 38654705728 for osd.2 2026-02-22T23:31:17.188 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:31:17.220 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:31:17.222 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:31:17.228 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:31:17.334 INFO:teuthology.orchestra.run.trial115.stdout:124554051623 2026-02-22T23:31:17.346 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051623 got 124554051623 for osd.3 2026-02-22T23:31:17.347 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:17.376 INFO:teuthology.orchestra.run.trial115.stdout:42949673026 2026-02-22T23:31:17.377 INFO:teuthology.orchestra.run.trial115.stdout:98784247864 2026-02-22T23:31:17.378 INFO:teuthology.orchestra.run.trial115.stdout:38654705729 2026-02-22T23:31:17.388 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949673026 got 42949673026 for osd.0 2026-02-22T23:31:17.388 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:17.389 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247864 got 98784247864 for osd.1 2026-02-22T23:31:17.390 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:17.391 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705729 got 38654705729 for osd.2 2026-02-22T23:31:17.391 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:17.391 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:31:17.576 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:17.577 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:31:17.586 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2026-02-22T23:31:16.162150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68602,"num_bytes_recovered":562291785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1393120,"kb_used_data":824092,"kb_used_omap":7,"kb_used_meta":568952,"kb_avail":281722400,"statfs":{"total":289910292480,"available":288483737600,"internally_reserved":0,"allocated":843870208,"data_stored":843944645,"data_compressed":17294,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":7633,"internal_metadata":582607407},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001227"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":205898,"reported_epoch":35,"state":"active+clean","last_fresh":"2026-02-22T23:31:06.902794+0000","last_change":"2026-02-22T23:31:06.902794+0000","last_active":"2026-02-22T23:31:06.902794+0000","last_peered":"2026-02-22T23:31:06.902794+0000","last_clean":"2026-02-22T23:31:06.902794+0000","last_became_active":"2026-02-22T23:31:06.902580+0000","last_became_peered":"2026-02-22T23:31:06.902580+0000","last_unstale":"2026-02-22T23:31:06.902794+0000","last_undegraded":"2026-02-22T23:31:06.902794+0000","last_fullsized":"2026-02-22T23:31:06.902794+0000","mapping_epoch":34,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:51:24.820525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68600,"num_bytes_recovered":561963593,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":61,"reported_epoch":35,"state":"active+clean","last_fresh":"2026-02-22T23:31:06.901992+0000","last_change":"2026-02-22T23:29:27.256843+0000","last_active":"2026-02-22T23:31:06.901992+0000","last_peered":"2026-02-22T23:31:06.901992+0000","last_clean":"2026-02-22T23:31:06.901992+0000","last_became_active":"2026-02-22T23:29:27.253029+0000","last_became_peered":"2026-02-22T23:29:27.253029+0000","last_unstale":"2026-02-22T23:31:06.901992+0000","last_undegraded":"2026-02-22T23:31:06.901992+0000","last_fullsized":"2026-02-22T23:31:06.901992+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:36:17.698676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68600,"num_bytes_recovered":561963593,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1122816000,"data_stored":1122848768,"data_compressed":20370,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":655872,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949673026,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":639016,"kb_used_data":274704,"kb_used_omap":1,"kb_used_meta":364286,"kb_avail":93732824,"statfs":{"total":96636764160,"available":95982411776,"internally_reserved":0,"allocated":281296896,"data_stored":281421719,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":373029318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051623,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313620,"kb_used_data":274684,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058220,"statfs":{"total":96636764160,"available":96315617280,"internally_reserved":0,"allocated":281276416,"data_stored":281097623,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440484,"kb_used_data":274704,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931356,"statfs":{"total":96636764160,"available":96185708544,"internally_reserved":0,"allocated":281296896,"data_stored":281425303,"data_compressed":7527,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":98784247864,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":279875584,"data_stored":279883776,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:31:17.587 DEBUG:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 205898, 'reported_epoch': 35, 'state': 'active+clean', 'last_fresh': '2026-02-22T23:31:06.902794+0000', 'last_change': '2026-02-22T23:31:06.902794+0000', 'last_active': '2026-02-22T23:31:06.902794+0000', 'last_peered': '2026-02-22T23:31:06.902794+0000', 'last_clean': '2026-02-22T23:31:06.902794+0000', 'last_became_active': '2026-02-22T23:31:06.902580+0000', 'last_became_peered': '2026-02-22T23:31:06.902580+0000', 'last_unstale': '2026-02-22T23:31:06.902794+0000', 'last_undegraded': '2026-02-22T23:31:06.902794+0000', 'last_fullsized': '2026-02-22T23:31:06.902794+0000', 'mapping_epoch': 34, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 35, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-24T07:51:24.820525+0000', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 68600, 'num_bytes_recovered': 561963593, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:31:17.588 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-22T23:31:17.764 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:17.764 INFO:teuthology.orchestra.run.trial115.stdout:{"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":233,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1771802962,"num_in_osds":3,"osd_in_since":1771802966,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":2}],"num_pgs":2,"num_pools":2,"num_objects":68601,"data_bytes":562283406,"bytes_used":1426554880,"bytes_avail":288483737600,"bytes_total":289910292480},"fsmap":{"epoch":1,"btime":"2026-02-22T23:27:24:013195+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-22T23:30:22.155712+0000","services":{}},"progress_events":{}} 2026-02-22T23:31:17.775 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:{'fsid': 'a4a3ada6-bd8c-4e08-a984-e18b05a50ce6', 'health': {'status': 'HEALTH_OK', 'checks': {}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 233, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 35, 'num_osds': 4, 'num_up_osds': 4, 'osd_up_since': 1771802962, 'num_in_osds': 3, 'osd_in_since': 1771802966, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 2}], 'num_pgs': 2, 'num_pools': 2, 'num_objects': 68601, 'data_bytes': 562283406, 'bytes_used': 1426554880, 'bytes_avail': 288483737600, 'bytes_total': 289910292480}, 'fsmap': {'epoch': 1, 'btime': '2026-02-22T23:27:24:013195+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 5, 'modified': '2026-02-22T23:30:22.155712+0000', 'services': {}}, 'progress_events': {}} 2026-02-22T23:31:17.775 DEBUG:tasks.ec_inconsistent_hinfo:there are 0 unfound objects 2026-02-22T23:31:17.775 INFO:tasks.ec_inconsistent_hinfo:remove test object hash info from non-primary osd.0 shard and test backfill 2026-02-22T23:31:17.775 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T23:31:17.924 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:17.925 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":35,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:31:06.894359+0000","last_up_change":"2026-02-22T23:29:22.235230+0000","last_in_change":"2026-02-22T23:29:26.244871+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T23:27:34.171905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-22T23:27:43.179120+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6809","nonce":3878013816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6810","nonce":3878013816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6812","nonce":3878013816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6811","nonce":3878013816}]},"public_addr":"10.20.193.115:6809/3878013816","cluster_addr":"10.20.193.115:6810/3878013816","heartbeat_back_addr":"10.20.193.115:6812/3878013816","heartbeat_front_addr":"10.20.193.115:6811/3878013816","state":["exists","up"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":23,"up_thru":26,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6805","nonce":2409177103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6806","nonce":2409177103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6808","nonce":2409177103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6807","nonce":2409177103}]},"public_addr":"10.20.193.115:6805/2409177103","cluster_addr":"10.20.193.115:6806/2409177103","heartbeat_back_addr":"10.20.193.115:6808/2409177103","heartbeat_front_addr":"10.20.193.115:6807/2409177103","state":["exists","up"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6801","nonce":1978117969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6802","nonce":1978117969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6804","nonce":1978117969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6803","nonce":1978117969}]},"public_addr":"10.20.193.115:6801/1978117969","cluster_addr":"10.20.193.115:6802/1978117969","heartbeat_back_addr":"10.20.193.115:6804/1978117969","heartbeat_front_addr":"10.20.193.115:6803/1978117969","state":["exists","up"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":25,"up_from":29,"up_thru":34,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6813","nonce":1033269798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6814","nonce":1033269798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6816","nonce":1033269798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6815","nonce":1033269798}]},"public_addr":"10.20.193.115:6813/1033269798","cluster_addr":"10.20.193.115:6814/1033269798","heartbeat_back_addr":"10.20.193.115:6816/1033269798","heartbeat_front_addr":"10.20.193.115:6815/1033269798","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:32.314887+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2026-02-22T23:27:31.613759+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:31.523737+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:32.017476+0000","dead_epoch":26}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:31:17.935 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json osd map unique_pool_0 test 2026-02-22T23:31:18.085 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:18.096 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":35,"pool":"unique_pool_0","pool_id":2,"objname":"test","raw_pgid":"2.40e8aab5","pgid":"2.0","up":[3,0,2],"up_primary":3,"acting":[3,0,2],"acting_primary":3} 2026-02-22T23:31:18.096 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-22T23:31:18.097 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T23:31:18.196 INFO:tasks.ceph.osd.0:Stopped 2026-02-22T23:31:18.197 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 2.0s1 --op list |grep '"oid":"test"') sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 2.0s1 "$object" rm-attr hinfo_key 2026-02-22T23:31:18.197 DEBUG:teuthology.orchestra.run.trial115:> bash -e -x -c 'object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 2.0s1 --op list |grep '"'"'"oid":"test"'"'"') 2026-02-22T23:31:18.197 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 2.0s1 "$object" rm-attr hinfo_key' 2026-02-22T23:31:18.201 INFO:teuthology.orchestra.run.trial115.stderr:+ '[' -z '' ']' 2026-02-22T23:31:18.201 INFO:teuthology.orchestra.run.trial115.stderr:+ return 2026-02-22T23:31:18.201 INFO:teuthology.orchestra.run.trial115.stderr:+ case $- in 2026-02-22T23:31:18.201 INFO:teuthology.orchestra.run.trial115.stderr:+ return 2026-02-22T23:31:18.202 INFO:teuthology.orchestra.run.trial115.stderr:++ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 2.0s1 --op list 2026-02-22T23:31:18.202 INFO:teuthology.orchestra.run.trial115.stderr:++ grep '"oid":"test"' 2026-02-22T23:31:20.589 INFO:teuthology.orchestra.run.trial115.stderr:+ object='["2.0s1",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":1,"max":0}]' 2026-02-22T23:31:20.589 INFO:teuthology.orchestra.run.trial115.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 2.0s1 '["2.0s1",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":1,"max":0}]' rm-attr hinfo_key 2026-02-22T23:31:24.981 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-22T23:31:24.981 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-02-22T23:31:25.024 INFO:tasks.ceph.osd.0:Started 2026-02-22T23:31:25.024 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-22T23:31:25.082 INFO:teuthology.orchestra.run.trial115.stderr:no valid command found; 10 closest matches: 2026-02-22T23:31:25.082 INFO:teuthology.orchestra.run.trial115.stderr:0 2026-02-22T23:31:25.083 INFO:teuthology.orchestra.run.trial115.stderr:1 2026-02-22T23:31:25.083 INFO:teuthology.orchestra.run.trial115.stderr:2 2026-02-22T23:31:25.083 INFO:teuthology.orchestra.run.trial115.stderr:abort 2026-02-22T23:31:25.083 INFO:teuthology.orchestra.run.trial115.stderr:assert 2026-02-22T23:31:25.083 INFO:teuthology.orchestra.run.trial115.stderr:config diff 2026-02-22T23:31:25.083 INFO:teuthology.orchestra.run.trial115.stderr:config diff get 2026-02-22T23:31:25.083 INFO:teuthology.orchestra.run.trial115.stderr:config get 2026-02-22T23:31:25.083 INFO:teuthology.orchestra.run.trial115.stderr:config help [] 2026-02-22T23:31:25.083 INFO:teuthology.orchestra.run.trial115.stderr:config set ... 2026-02-22T23:31:25.083 INFO:teuthology.orchestra.run.trial115.stderr:admin_socket: invalid command 2026-02-22T23:31:25.085 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-22T23:31:25.085 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-22T23:31:26.296 INFO:tasks.ceph.osd.0.trial115.stderr:2026-02-22T23:31:26.294+0000 7f086b9508c0 -1 Falling back to public interface 2026-02-22T23:31:30.086 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-22T23:31:30.146 INFO:teuthology.orchestra.run.trial115.stderr:no valid command found; 10 closest matches: 2026-02-22T23:31:30.147 INFO:teuthology.orchestra.run.trial115.stderr:0 2026-02-22T23:31:30.147 INFO:teuthology.orchestra.run.trial115.stderr:1 2026-02-22T23:31:30.147 INFO:teuthology.orchestra.run.trial115.stderr:2 2026-02-22T23:31:30.147 INFO:teuthology.orchestra.run.trial115.stderr:abort 2026-02-22T23:31:30.147 INFO:teuthology.orchestra.run.trial115.stderr:assert 2026-02-22T23:31:30.147 INFO:teuthology.orchestra.run.trial115.stderr:config diff 2026-02-22T23:31:30.147 INFO:teuthology.orchestra.run.trial115.stderr:config diff get 2026-02-22T23:31:30.147 INFO:teuthology.orchestra.run.trial115.stderr:config get 2026-02-22T23:31:30.148 INFO:teuthology.orchestra.run.trial115.stderr:config help [] 2026-02-22T23:31:30.148 INFO:teuthology.orchestra.run.trial115.stderr:config set ... 2026-02-22T23:31:30.148 INFO:teuthology.orchestra.run.trial115.stderr:admin_socket: invalid command 2026-02-22T23:31:30.149 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-22T23:31:30.149 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-22T23:31:30.626 INFO:tasks.ceph.osd.0.trial115.stderr:2026-02-22T23:31:30.622+0000 7f086b9508c0 -1 osd.0 35 log_to_monitors true 2026-02-22T23:31:31.253 INFO:tasks.ceph.osd.0.trial115.stderr:2026-02-22T23:31:31.250+0000 7f085f72c640 -1 osd.0 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T23:31:35.150 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-22T23:31:35.223 INFO:teuthology.orchestra.run.trial115.stdout:{ 2026-02-22T23:31:35.223 INFO:teuthology.orchestra.run.trial115.stdout: "ops": [], 2026-02-22T23:31:35.224 INFO:teuthology.orchestra.run.trial115.stdout: "num_ops": 0 2026-02-22T23:31:35.224 INFO:teuthology.orchestra.run.trial115.stdout:} 2026-02-22T23:31:35.231 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for osd.0 to be up 2026-02-22T23:31:35.231 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T23:31:35.376 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:35.376 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":40,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:31:33.252247+0000","last_up_change":"2026-02-22T23:31:32.251044+0000","last_in_change":"2026-02-22T23:29:26.244871+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T23:27:34.171905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-22T23:27:43.179120+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":35,"up_from":39,"up_thru":39,"down_at":36,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6809","nonce":2954119946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6810","nonce":2954119946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6812","nonce":2954119946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6811","nonce":2954119946}]},"public_addr":"10.20.193.115:6809/2954119946","cluster_addr":"10.20.193.115:6810/2954119946","heartbeat_back_addr":"10.20.193.115:6812/2954119946","heartbeat_front_addr":"10.20.193.115:6811/2954119946","state":["exists","up"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":23,"up_thru":26,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6805","nonce":2409177103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6806","nonce":2409177103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6808","nonce":2409177103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6807","nonce":2409177103}]},"public_addr":"10.20.193.115:6805/2409177103","cluster_addr":"10.20.193.115:6806/2409177103","heartbeat_back_addr":"10.20.193.115:6808/2409177103","heartbeat_front_addr":"10.20.193.115:6807/2409177103","state":["exists","up"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6801","nonce":1978117969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6802","nonce":1978117969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6804","nonce":1978117969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6803","nonce":1978117969}]},"public_addr":"10.20.193.115:6801/1978117969","cluster_addr":"10.20.193.115:6802/1978117969","heartbeat_back_addr":"10.20.193.115:6804/1978117969","heartbeat_front_addr":"10.20.193.115:6803/1978117969","state":["exists","up"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":25,"up_from":29,"up_thru":39,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6813","nonce":1033269798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6814","nonce":1033269798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6816","nonce":1033269798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6815","nonce":1033269798}]},"public_addr":"10.20.193.115:6813/1033269798","cluster_addr":"10.20.193.115:6814/1033269798","heartbeat_back_addr":"10.20.193.115:6816/1033269798","heartbeat_front_addr":"10.20.193.115:6815/1033269798","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:32.314887+0000","dead_epoch":36},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2026-02-22T23:27:31.613759+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:31.523737+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:32.017476+0000","dead_epoch":26}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:31:35.387 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:osd.0 is up 2026-02-22T23:31:35.388 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2026-02-22T23:31:35.542 INFO:tasks.ceph.mon.a.trial115.stderr:2026-02-22T23:31:35.538+0000 7f2aeaf18640 -1 mon.a@0(leader).osd e40 definitely_dead 0 2026-02-22T23:31:36.431 INFO:teuthology.orchestra.run.trial115.stderr:marked in osd.1. 2026-02-22T23:31:36.445 DEBUG:tasks.ec_inconsistent_hinfo:waiting for pg 2.0 backfill from osd.0 to osd.1 complete 2026-02-22T23:31:41.446 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:31:41.446 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:31:41.446 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:31:41.446 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:31:41.536 INFO:teuthology.orchestra.run.trial115.stdout:38654705735 2026-02-22T23:31:41.536 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:31:41.538 INFO:teuthology.orchestra.run.trial115.stdout:98784247870 2026-02-22T23:31:41.538 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:31:41.539 INFO:teuthology.orchestra.run.trial115.stdout:167503724548 2026-02-22T23:31:41.539 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:31:41.568 INFO:teuthology.orchestra.run.trial115.stdout:124554051630 2026-02-22T23:31:41.568 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:31:41.689 INFO:teuthology.orchestra.run.trial115.stdout:38654705734 2026-02-22T23:31:41.702 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705735 got 38654705734 for osd.2 2026-02-22T23:31:41.705 INFO:teuthology.orchestra.run.trial115.stdout:167503724546 2026-02-22T23:31:41.714 INFO:teuthology.orchestra.run.trial115.stdout:98784247869 2026-02-22T23:31:41.717 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724548 got 167503724546 for osd.0 2026-02-22T23:31:41.727 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247870 got 98784247869 for osd.1 2026-02-22T23:31:41.758 INFO:teuthology.orchestra.run.trial115.stdout:124554051628 2026-02-22T23:31:41.772 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051630 got 124554051628 for osd.3 2026-02-22T23:31:42.703 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:31:42.718 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:31:42.727 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:31:42.773 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:31:42.849 INFO:teuthology.orchestra.run.trial115.stdout:38654705735 2026-02-22T23:31:42.862 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705735 got 38654705735 for osd.2 2026-02-22T23:31:42.862 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:42.896 INFO:teuthology.orchestra.run.trial115.stdout:167503724548 2026-02-22T23:31:42.898 INFO:teuthology.orchestra.run.trial115.stdout:98784247870 2026-02-22T23:31:42.908 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724548 got 167503724548 for osd.0 2026-02-22T23:31:42.909 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:42.910 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247870 got 98784247870 for osd.1 2026-02-22T23:31:42.910 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:42.984 INFO:teuthology.orchestra.run.trial115.stdout:124554051630 2026-02-22T23:31:43.000 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051630 got 124554051630 for osd.3 2026-02-22T23:31:43.001 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:43.001 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:31:43.145 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:43.146 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:31:43.156 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2026-02-22T23:31:42.165671+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69080,"num_bytes_recovered":566207049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1500212,"kb_used_data":1099028,"kb_used_omap":14,"kb_used_meta":401073,"kb_avail":375987148,"statfs":{"total":386547056640,"available":385010839552,"internally_reserved":0,"allocated":1125404672,"data_stored":1125153620,"data_compressed":21556,"data_compressed_allocated":73728,"data_compressed_original":720896,"omap_allocated":14837,"internal_metadata":410699275},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-68601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":480,"num_bytes_recovered":4243456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000797"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":206886,"reported_epoch":43,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:31:39.183926+0000","last_change":"2026-02-22T23:31:38.439068+0000","last_active":"2026-02-22T23:31:39.183926+0000","last_peered":"2026-02-22T23:31:39.183926+0000","last_clean":"2026-02-22T23:31:33.258159+0000","last_became_active":"2026-02-22T23:31:38.438327+0000","last_became_peered":"2026-02-22T23:31:38.438327+0000","last_unstale":"2026-02-22T23:31:39.183926+0000","last_undegraded":"2026-02-22T23:31:39.183926+0000","last_fullsized":"2026-02-22T23:31:39.183926+0000","mapping_epoch":42,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69078,"num_bytes_recovered":565879369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,0,2],"avail_no_missing":["3(0)","0(1)","1(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":18,"reported_epoch":43,"state":"active+clean","last_fresh":"2026-02-22T23:31:38.437517+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:31:38.437517+0000","last_peered":"2026-02-22T23:31:38.437517+0000","last_clean":"2026-02-22T23:31:38.437517+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:31:38.437517+0000","last_undegraded":"2026-02-22T23:31:38.437517+0000","last_fullsized":"2026-02-22T23:31:38.437517+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69078,"num_bytes_recovered":565879369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123917824,"data_stored":1123950592,"data_compressed":20370,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724548,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313192,"kb_used_data":274768,"kb_used_omap":4,"kb_used_meta":38395,"kb_avail":94058648,"statfs":{"total":96636764160,"available":96316055552,"internally_reserved":0,"allocated":281362432,"data_stored":281450581,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4401,"internal_metadata":39317199},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051630,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313688,"kb_used_data":274748,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058152,"statfs":{"total":96636764160,"available":96315547648,"internally_reserved":0,"allocated":281341952,"data_stored":281126485,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440532,"kb_used_data":274748,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931308,"statfs":{"total":96636764160,"available":96185659392,"internally_reserved":0,"allocated":281341952,"data_stored":281126485,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247870,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":432800,"kb_used_data":274764,"kb_used_omap":4,"kb_used_meta":158011,"kb_avail":93939040,"statfs":{"total":96636764160,"available":96193576960,"internally_reserved":0,"allocated":281358336,"data_stored":281450069,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":161803987},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:31:43.157 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 206886, 'reported_epoch': 43, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:31:39.183926+0000', 'last_change': '2026-02-22T23:31:38.439068+0000', 'last_active': '2026-02-22T23:31:39.183926+0000', 'last_peered': '2026-02-22T23:31:39.183926+0000', 'last_clean': '2026-02-22T23:31:33.258159+0000', 'last_became_active': '2026-02-22T23:31:38.438327+0000', 'last_became_peered': '2026-02-22T23:31:38.438327+0000', 'last_unstale': '2026-02-22T23:31:39.183926+0000', 'last_undegraded': '2026-02-22T23:31:39.183926+0000', 'last_fullsized': '2026-02-22T23:31:39.183926+0000', 'mapping_epoch': 42, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 40, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 69078, 'num_bytes_recovered': 565879369, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 2], 'acting': [3, 0, 2], 'avail_no_missing': ['3(0)', '0(1)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:31:43.157 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:31:48.159 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:31:48.159 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:31:48.160 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:31:48.160 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:31:48.245 INFO:teuthology.orchestra.run.trial115.stdout:167503724550 2026-02-22T23:31:48.245 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:31:48.246 INFO:teuthology.orchestra.run.trial115.stdout:38654705737 2026-02-22T23:31:48.247 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:31:48.249 INFO:teuthology.orchestra.run.trial115.stdout:98784247873 2026-02-22T23:31:48.250 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:31:48.269 INFO:teuthology.orchestra.run.trial115.stdout:124554051632 2026-02-22T23:31:48.269 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:31:48.392 INFO:teuthology.orchestra.run.trial115.stdout:167503724549 2026-02-22T23:31:48.401 INFO:teuthology.orchestra.run.trial115.stdout:98784247871 2026-02-22T23:31:48.404 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724550 got 167503724549 for osd.0 2026-02-22T23:31:48.412 INFO:teuthology.orchestra.run.trial115.stdout:38654705736 2026-02-22T23:31:48.413 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247873 got 98784247871 for osd.1 2026-02-22T23:31:48.425 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705737 got 38654705736 for osd.2 2026-02-22T23:31:48.465 INFO:teuthology.orchestra.run.trial115.stdout:124554051631 2026-02-22T23:31:48.478 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051632 got 124554051631 for osd.3 2026-02-22T23:31:49.406 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:31:49.414 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:31:49.426 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:31:49.478 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:31:49.555 INFO:teuthology.orchestra.run.trial115.stdout:167503724549 2026-02-22T23:31:49.567 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724550 got 167503724549 for osd.0 2026-02-22T23:31:49.599 INFO:teuthology.orchestra.run.trial115.stdout:38654705736 2026-02-22T23:31:49.607 INFO:teuthology.orchestra.run.trial115.stdout:98784247871 2026-02-22T23:31:49.611 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705737 got 38654705736 for osd.2 2026-02-22T23:31:49.619 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247873 got 98784247871 for osd.1 2026-02-22T23:31:49.652 INFO:teuthology.orchestra.run.trial115.stdout:124554051631 2026-02-22T23:31:49.663 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051632 got 124554051631 for osd.3 2026-02-22T23:31:50.568 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:31:50.612 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:31:50.621 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:31:50.664 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:31:50.712 INFO:teuthology.orchestra.run.trial115.stdout:167503724550 2026-02-22T23:31:50.724 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724550 got 167503724550 for osd.0 2026-02-22T23:31:50.724 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:50.759 INFO:teuthology.orchestra.run.trial115.stdout:38654705738 2026-02-22T23:31:50.771 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705737 got 38654705738 for osd.2 2026-02-22T23:31:50.772 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:50.800 INFO:teuthology.orchestra.run.trial115.stdout:98784247873 2026-02-22T23:31:50.811 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247873 got 98784247873 for osd.1 2026-02-22T23:31:50.812 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:50.854 INFO:teuthology.orchestra.run.trial115.stdout:124554051632 2026-02-22T23:31:50.867 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051632 got 124554051632 for osd.3 2026-02-22T23:31:50.867 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:50.868 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:31:51.013 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:51.013 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:31:51.025 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2026-02-22T23:31:50.252446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69080,"num_bytes_recovered":566207049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1518256,"kb_used_data":1099028,"kb_used_omap":14,"kb_used_meta":419121,"kb_avail":375969104,"statfs":{"total":386547056640,"available":384992362496,"internally_reserved":0,"allocated":1125404672,"data_stored":1125153620,"data_compressed":21556,"data_compressed_allocated":73728,"data_compressed_original":720896,"omap_allocated":14837,"internal_metadata":429180427},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":478,"num_bytes_recovered":3915776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.087013"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":206886,"reported_epoch":43,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:31:39.183926+0000","last_change":"2026-02-22T23:31:38.439068+0000","last_active":"2026-02-22T23:31:39.183926+0000","last_peered":"2026-02-22T23:31:39.183926+0000","last_clean":"2026-02-22T23:31:33.258159+0000","last_became_active":"2026-02-22T23:31:38.438327+0000","last_became_peered":"2026-02-22T23:31:38.438327+0000","last_unstale":"2026-02-22T23:31:39.183926+0000","last_undegraded":"2026-02-22T23:31:39.183926+0000","last_fullsized":"2026-02-22T23:31:39.183926+0000","mapping_epoch":42,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69078,"num_bytes_recovered":565879369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,0,2],"avail_no_missing":["3(0)","0(1)","1(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":18,"reported_epoch":43,"state":"active+clean","last_fresh":"2026-02-22T23:31:38.437517+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:31:38.437517+0000","last_peered":"2026-02-22T23:31:38.437517+0000","last_clean":"2026-02-22T23:31:38.437517+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:31:38.437517+0000","last_undegraded":"2026-02-22T23:31:38.437517+0000","last_fullsized":"2026-02-22T23:31:38.437517+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69078,"num_bytes_recovered":565879369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123917824,"data_stored":1123950592,"data_compressed":20370,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724550,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313192,"kb_used_data":274768,"kb_used_omap":4,"kb_used_meta":38395,"kb_avail":94058648,"statfs":{"total":96636764160,"available":96316055552,"internally_reserved":0,"allocated":281362432,"data_stored":281450581,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4401,"internal_metadata":39317199},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051632,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313688,"kb_used_data":274748,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058152,"statfs":{"total":96636764160,"available":96315547648,"internally_reserved":0,"allocated":281341952,"data_stored":281126485,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440532,"kb_used_data":274748,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931308,"statfs":{"total":96636764160,"available":96185659392,"internally_reserved":0,"allocated":281341952,"data_stored":281126485,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450844,"kb_used_data":274764,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920996,"statfs":{"total":96636764160,"available":96175099904,"internally_reserved":0,"allocated":281358336,"data_stored":281450069,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:31:51.026 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 206886, 'reported_epoch': 43, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:31:39.183926+0000', 'last_change': '2026-02-22T23:31:38.439068+0000', 'last_active': '2026-02-22T23:31:39.183926+0000', 'last_peered': '2026-02-22T23:31:39.183926+0000', 'last_clean': '2026-02-22T23:31:33.258159+0000', 'last_became_active': '2026-02-22T23:31:38.438327+0000', 'last_became_peered': '2026-02-22T23:31:38.438327+0000', 'last_unstale': '2026-02-22T23:31:39.183926+0000', 'last_undegraded': '2026-02-22T23:31:39.183926+0000', 'last_fullsized': '2026-02-22T23:31:39.183926+0000', 'mapping_epoch': 42, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 40, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 69078, 'num_bytes_recovered': 565879369, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 2], 'acting': [3, 0, 2], 'avail_no_missing': ['3(0)', '0(1)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:31:51.026 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:31:56.028 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:31:56.028 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:31:56.029 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:31:56.029 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:31:56.113 INFO:teuthology.orchestra.run.trial115.stdout:124554051634 2026-02-22T23:31:56.114 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:31:56.115 INFO:teuthology.orchestra.run.trial115.stdout:167503724552 2026-02-22T23:31:56.116 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:31:56.117 INFO:teuthology.orchestra.run.trial115.stdout:38654705740 2026-02-22T23:31:56.118 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:31:56.119 INFO:teuthology.orchestra.run.trial115.stdout:98784247875 2026-02-22T23:31:56.119 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:31:56.265 INFO:teuthology.orchestra.run.trial115.stdout:124554051634 2026-02-22T23:31:56.279 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051634 got 124554051634 for osd.3 2026-02-22T23:31:56.279 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:56.301 INFO:teuthology.orchestra.run.trial115.stdout:167503724552 2026-02-22T23:31:56.310 INFO:teuthology.orchestra.run.trial115.stdout:38654705740 2026-02-22T23:31:56.313 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724552 got 167503724552 for osd.0 2026-02-22T23:31:56.313 INFO:teuthology.orchestra.run.trial115.stdout:98784247875 2026-02-22T23:31:56.313 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:56.322 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705740 got 38654705740 for osd.2 2026-02-22T23:31:56.322 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:56.324 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247875 got 98784247875 for osd.1 2026-02-22T23:31:56.324 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:56.324 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:31:56.512 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:56.512 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:31:56.522 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2026-02-22T23:31:56.167543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69080,"num_bytes_recovered":566207049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1252236,"kb_used_data":824700,"kb_used_omap":14,"kb_used_meta":427441,"kb_avail":376235124,"statfs":{"total":386547056640,"available":385264766976,"internally_reserved":0,"allocated":844492800,"data_stored":844195616,"data_compressed":17294,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":14837,"internal_metadata":437700107},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001146"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":206899,"reported_epoch":45,"state":"active+clean","last_fresh":"2026-02-22T23:31:51.257494+0000","last_change":"2026-02-22T23:31:51.257494+0000","last_active":"2026-02-22T23:31:51.257494+0000","last_peered":"2026-02-22T23:31:51.257494+0000","last_clean":"2026-02-22T23:31:51.257494+0000","last_became_active":"2026-02-22T23:31:51.257186+0000","last_became_peered":"2026-02-22T23:31:51.257186+0000","last_unstale":"2026-02-22T23:31:51.257494+0000","last_undegraded":"2026-02-22T23:31:51.257494+0000","last_fullsized":"2026-02-22T23:31:51.257494+0000","mapping_epoch":44,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:57:22.463415+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69078,"num_bytes_recovered":565879369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":22,"reported_epoch":45,"state":"active+clean","last_fresh":"2026-02-22T23:31:51.256552+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:31:51.256552+0000","last_peered":"2026-02-22T23:31:51.256552+0000","last_clean":"2026-02-22T23:31:51.256552+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:31:51.256552+0000","last_undegraded":"2026-02-22T23:31:51.256552+0000","last_fullsized":"2026-02-22T23:31:51.256552+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69078,"num_bytes_recovered":565879369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123917824,"data_stored":1123950592,"data_compressed":20370,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47128,"kb_used_data":392,"kb_used_omap":4,"kb_used_meta":46715,"kb_avail":94324712,"statfs":{"total":96636764160,"available":96588505088,"internally_reserved":0,"allocated":401408,"data_stored":471880,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":4401,"internal_metadata":47836879},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051634,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313696,"kb_used_data":274764,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058144,"statfs":{"total":96636764160,"available":96315539456,"internally_reserved":0,"allocated":281358336,"data_stored":281133384,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440548,"kb_used_data":274764,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931292,"statfs":{"total":96636764160,"available":96185643008,"internally_reserved":0,"allocated":281358336,"data_stored":281133384,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247875,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450864,"kb_used_data":274780,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920976,"statfs":{"total":96636764160,"available":96175079424,"internally_reserved":0,"allocated":281374720,"data_stored":281456968,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:31:56.523 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 206899, 'reported_epoch': 45, 'state': 'active+clean', 'last_fresh': '2026-02-22T23:31:51.257494+0000', 'last_change': '2026-02-22T23:31:51.257494+0000', 'last_active': '2026-02-22T23:31:51.257494+0000', 'last_peered': '2026-02-22T23:31:51.257494+0000', 'last_clean': '2026-02-22T23:31:51.257494+0000', 'last_became_active': '2026-02-22T23:31:51.257186+0000', 'last_became_peered': '2026-02-22T23:31:51.257186+0000', 'last_unstale': '2026-02-22T23:31:51.257494+0000', 'last_undegraded': '2026-02-22T23:31:51.257494+0000', 'last_fullsized': '2026-02-22T23:31:51.257494+0000', 'mapping_epoch': 44, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 45, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-24T01:57:22.463415+0000', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 69078, 'num_bytes_recovered': 565879369, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 2], 'acting': [3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:31:56.523 DEBUG:tasks.ec_inconsistent_hinfo:backfilled! 2026-02-22T23:31:56.523 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for clean 2026-02-22T23:31:56.523 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:31:56.664 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:56.664 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:31:56.674 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2026-02-22T23:31:56.167543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69080,"num_bytes_recovered":566207049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1252236,"kb_used_data":824700,"kb_used_omap":14,"kb_used_meta":427441,"kb_avail":376235124,"statfs":{"total":386547056640,"available":385264766976,"internally_reserved":0,"allocated":844492800,"data_stored":844195616,"data_compressed":17294,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":14837,"internal_metadata":437700107},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001146"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":206899,"reported_epoch":45,"state":"active+clean","last_fresh":"2026-02-22T23:31:51.257494+0000","last_change":"2026-02-22T23:31:51.257494+0000","last_active":"2026-02-22T23:31:51.257494+0000","last_peered":"2026-02-22T23:31:51.257494+0000","last_clean":"2026-02-22T23:31:51.257494+0000","last_became_active":"2026-02-22T23:31:51.257186+0000","last_became_peered":"2026-02-22T23:31:51.257186+0000","last_unstale":"2026-02-22T23:31:51.257494+0000","last_undegraded":"2026-02-22T23:31:51.257494+0000","last_fullsized":"2026-02-22T23:31:51.257494+0000","mapping_epoch":44,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:57:22.463415+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69078,"num_bytes_recovered":565879369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":22,"reported_epoch":45,"state":"active+clean","last_fresh":"2026-02-22T23:31:51.256552+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:31:51.256552+0000","last_peered":"2026-02-22T23:31:51.256552+0000","last_clean":"2026-02-22T23:31:51.256552+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:31:51.256552+0000","last_undegraded":"2026-02-22T23:31:51.256552+0000","last_fullsized":"2026-02-22T23:31:51.256552+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69078,"num_bytes_recovered":565879369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123917824,"data_stored":1123950592,"data_compressed":20370,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47128,"kb_used_data":392,"kb_used_omap":4,"kb_used_meta":46715,"kb_avail":94324712,"statfs":{"total":96636764160,"available":96588505088,"internally_reserved":0,"allocated":401408,"data_stored":471880,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":4401,"internal_metadata":47836879},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051634,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313696,"kb_used_data":274764,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058144,"statfs":{"total":96636764160,"available":96315539456,"internally_reserved":0,"allocated":281358336,"data_stored":281133384,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440548,"kb_used_data":274764,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931292,"statfs":{"total":96636764160,"available":96185643008,"internally_reserved":0,"allocated":281358336,"data_stored":281133384,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247875,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450864,"kb_used_data":274780,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920976,"statfs":{"total":96636764160,"available":96175079424,"internally_reserved":0,"allocated":281374720,"data_stored":281456968,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:31:56.675 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:31:56.817 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:56.817 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:31:56.827 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2026-02-22T23:31:56.167543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69080,"num_bytes_recovered":566207049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1252236,"kb_used_data":824700,"kb_used_omap":14,"kb_used_meta":427441,"kb_avail":376235124,"statfs":{"total":386547056640,"available":385264766976,"internally_reserved":0,"allocated":844492800,"data_stored":844195616,"data_compressed":17294,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":14837,"internal_metadata":437700107},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001146"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":206899,"reported_epoch":45,"state":"active+clean","last_fresh":"2026-02-22T23:31:51.257494+0000","last_change":"2026-02-22T23:31:51.257494+0000","last_active":"2026-02-22T23:31:51.257494+0000","last_peered":"2026-02-22T23:31:51.257494+0000","last_clean":"2026-02-22T23:31:51.257494+0000","last_became_active":"2026-02-22T23:31:51.257186+0000","last_became_peered":"2026-02-22T23:31:51.257186+0000","last_unstale":"2026-02-22T23:31:51.257494+0000","last_undegraded":"2026-02-22T23:31:51.257494+0000","last_fullsized":"2026-02-22T23:31:51.257494+0000","mapping_epoch":44,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:57:22.463415+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69078,"num_bytes_recovered":565879369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":22,"reported_epoch":45,"state":"active+clean","last_fresh":"2026-02-22T23:31:51.256552+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:31:51.256552+0000","last_peered":"2026-02-22T23:31:51.256552+0000","last_clean":"2026-02-22T23:31:51.256552+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:31:51.256552+0000","last_undegraded":"2026-02-22T23:31:51.256552+0000","last_fullsized":"2026-02-22T23:31:51.256552+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69078,"num_bytes_recovered":565879369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123917824,"data_stored":1123950592,"data_compressed":20370,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47128,"kb_used_data":392,"kb_used_omap":4,"kb_used_meta":46715,"kb_avail":94324712,"statfs":{"total":96636764160,"available":96588505088,"internally_reserved":0,"allocated":401408,"data_stored":471880,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":4401,"internal_metadata":47836879},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051634,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313696,"kb_used_data":274764,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058144,"statfs":{"total":96636764160,"available":96315539456,"internally_reserved":0,"allocated":281358336,"data_stored":281133384,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440548,"kb_used_data":274764,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931292,"statfs":{"total":96636764160,"available":96185643008,"internally_reserved":0,"allocated":281358336,"data_stored":281133384,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247875,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450864,"kb_used_data":274780,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920976,"statfs":{"total":96636764160,"available":96175079424,"internally_reserved":0,"allocated":281374720,"data_stored":281456968,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:31:56.828 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:clean! 2026-02-22T23:31:56.828 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:31:56.829 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:31:56.830 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:31:56.830 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:31:56.915 INFO:teuthology.orchestra.run.trial115.stdout:167503724554 2026-02-22T23:31:56.916 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:31:56.917 INFO:teuthology.orchestra.run.trial115.stdout:124554051636 2026-02-22T23:31:56.917 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:31:56.918 INFO:teuthology.orchestra.run.trial115.stdout:38654705741 2026-02-22T23:31:56.918 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:31:56.921 INFO:teuthology.orchestra.run.trial115.stdout:98784247876 2026-02-22T23:31:56.922 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:31:57.109 INFO:teuthology.orchestra.run.trial115.stdout:167503724552 2026-02-22T23:31:57.111 INFO:teuthology.orchestra.run.trial115.stdout:38654705740 2026-02-22T23:31:57.118 INFO:teuthology.orchestra.run.trial115.stdout:124554051634 2026-02-22T23:31:57.124 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705741 got 38654705740 for osd.2 2026-02-22T23:31:57.124 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724554 got 167503724552 for osd.0 2026-02-22T23:31:57.131 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051636 got 124554051634 for osd.3 2026-02-22T23:31:57.158 INFO:teuthology.orchestra.run.trial115.stdout:98784247875 2026-02-22T23:31:57.170 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247876 got 98784247875 for osd.1 2026-02-22T23:31:58.125 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:31:58.125 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:31:58.132 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:31:58.170 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:31:58.272 INFO:teuthology.orchestra.run.trial115.stdout:167503724554 2026-02-22T23:31:58.277 INFO:teuthology.orchestra.run.trial115.stdout:38654705741 2026-02-22T23:31:58.284 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724554 got 167503724554 for osd.0 2026-02-22T23:31:58.284 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:58.292 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705741 got 38654705741 for osd.2 2026-02-22T23:31:58.292 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:58.313 INFO:teuthology.orchestra.run.trial115.stdout:124554051636 2026-02-22T23:31:58.322 INFO:teuthology.orchestra.run.trial115.stdout:98784247876 2026-02-22T23:31:58.325 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 124554051636 got 124554051636 for osd.3 2026-02-22T23:31:58.325 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:58.334 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247876 got 98784247876 for osd.1 2026-02-22T23:31:58.334 DEBUG:teuthology.parallel:result is None 2026-02-22T23:31:58.335 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:31:58.523 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:58.524 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:31:58.534 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2026-02-22T23:31:58.167750+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69080,"num_bytes_recovered":566207049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1252236,"kb_used_data":824700,"kb_used_omap":14,"kb_used_meta":427441,"kb_avail":376235124,"statfs":{"total":386547056640,"available":385264766976,"internally_reserved":0,"allocated":844492800,"data_stored":844195616,"data_compressed":17294,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":14837,"internal_metadata":437700107},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001046"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":206899,"reported_epoch":45,"state":"active+clean","last_fresh":"2026-02-22T23:31:51.257494+0000","last_change":"2026-02-22T23:31:51.257494+0000","last_active":"2026-02-22T23:31:51.257494+0000","last_peered":"2026-02-22T23:31:51.257494+0000","last_clean":"2026-02-22T23:31:51.257494+0000","last_became_active":"2026-02-22T23:31:51.257186+0000","last_became_peered":"2026-02-22T23:31:51.257186+0000","last_unstale":"2026-02-22T23:31:51.257494+0000","last_undegraded":"2026-02-22T23:31:51.257494+0000","last_fullsized":"2026-02-22T23:31:51.257494+0000","mapping_epoch":44,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T01:57:22.463415+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69078,"num_bytes_recovered":565879369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":22,"reported_epoch":45,"state":"active+clean","last_fresh":"2026-02-22T23:31:51.256552+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:31:51.256552+0000","last_peered":"2026-02-22T23:31:51.256552+0000","last_clean":"2026-02-22T23:31:51.256552+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:31:51.256552+0000","last_undegraded":"2026-02-22T23:31:51.256552+0000","last_fullsized":"2026-02-22T23:31:51.256552+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69078,"num_bytes_recovered":565879369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123917824,"data_stored":1123950592,"data_compressed":20370,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724554,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47128,"kb_used_data":392,"kb_used_omap":4,"kb_used_meta":46715,"kb_avail":94324712,"statfs":{"total":96636764160,"available":96588505088,"internally_reserved":0,"allocated":401408,"data_stored":471880,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":4401,"internal_metadata":47836879},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":29,"seq":124554051636,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313696,"kb_used_data":274764,"kb_used_omap":4,"kb_used_meta":38907,"kb_avail":94058144,"statfs":{"total":96636764160,"available":96315539456,"internally_reserved":0,"allocated":281358336,"data_stored":281133384,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4445,"internal_metadata":39841443},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440548,"kb_used_data":274764,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93931292,"statfs":{"total":96636764160,"available":96185643008,"internally_reserved":0,"allocated":281358336,"data_stored":281133384,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247876,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450864,"kb_used_data":274780,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920976,"statfs":{"total":96636764160,"available":96175079424,"internally_reserved":0,"allocated":281374720,"data_stored":281456968,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:31:58.535 DEBUG:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 206899, 'reported_epoch': 45, 'state': 'active+clean', 'last_fresh': '2026-02-22T23:31:51.257494+0000', 'last_change': '2026-02-22T23:31:51.257494+0000', 'last_active': '2026-02-22T23:31:51.257494+0000', 'last_peered': '2026-02-22T23:31:51.257494+0000', 'last_clean': '2026-02-22T23:31:51.257494+0000', 'last_became_active': '2026-02-22T23:31:51.257186+0000', 'last_became_peered': '2026-02-22T23:31:51.257186+0000', 'last_unstale': '2026-02-22T23:31:51.257494+0000', 'last_undegraded': '2026-02-22T23:31:51.257494+0000', 'last_fullsized': '2026-02-22T23:31:51.257494+0000', 'mapping_epoch': 44, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 45, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-24T01:57:22.463415+0000', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 69078, 'num_bytes_recovered': 565879369, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 2], 'acting': [3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:31:58.535 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-22T23:31:58.710 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:58.710 INFO:teuthology.orchestra.run.trial115.stdout:{"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":274,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1771803092,"num_in_osds":4,"osd_in_since":1771803096,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":2}],"num_pgs":2,"num_pools":2,"num_objects":68601,"data_bytes":562283406,"bytes_used":1282289664,"bytes_avail":385264766976,"bytes_total":386547056640},"fsmap":{"epoch":1,"btime":"2026-02-22T23:27:24:013195+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-22T23:30:22.155712+0000","services":{}},"progress_events":{}} 2026-02-22T23:31:58.722 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:{'fsid': 'a4a3ada6-bd8c-4e08-a984-e18b05a50ce6', 'health': {'status': 'HEALTH_OK', 'checks': {}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 274, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 45, 'num_osds': 4, 'num_up_osds': 4, 'osd_up_since': 1771803092, 'num_in_osds': 4, 'osd_in_since': 1771803096, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 2}], 'num_pgs': 2, 'num_pools': 2, 'num_objects': 68601, 'data_bytes': 562283406, 'bytes_used': 1282289664, 'bytes_avail': 385264766976, 'bytes_total': 386547056640}, 'fsmap': {'epoch': 1, 'btime': '2026-02-22T23:27:24:013195+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 5, 'modified': '2026-02-22T23:30:22.155712+0000', 'services': {}}, 'progress_events': {}} 2026-02-22T23:31:58.722 DEBUG:tasks.ec_inconsistent_hinfo:there are 0 unfound objects 2026-02-22T23:31:58.722 INFO:tasks.ec_inconsistent_hinfo:remove hash info from two shards and test backfill 2026-02-22T23:31:58.722 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T23:31:58.877 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:58.877 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":45,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:31:51.251835+0000","last_up_change":"2026-02-22T23:31:32.251044+0000","last_in_change":"2026-02-22T23:31:36.429779+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T23:27:34.171905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-22T23:27:43.179120+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":35,"up_from":39,"up_thru":39,"down_at":36,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6809","nonce":2954119946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6810","nonce":2954119946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6812","nonce":2954119946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6811","nonce":2954119946}]},"public_addr":"10.20.193.115:6809/2954119946","cluster_addr":"10.20.193.115:6810/2954119946","heartbeat_back_addr":"10.20.193.115:6812/2954119946","heartbeat_front_addr":"10.20.193.115:6811/2954119946","state":["exists","up"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":23,"up_thru":41,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6805","nonce":2409177103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6806","nonce":2409177103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6808","nonce":2409177103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6807","nonce":2409177103}]},"public_addr":"10.20.193.115:6805/2409177103","cluster_addr":"10.20.193.115:6806/2409177103","heartbeat_back_addr":"10.20.193.115:6808/2409177103","heartbeat_front_addr":"10.20.193.115:6807/2409177103","state":["exists","up"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6801","nonce":1978117969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6802","nonce":1978117969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6804","nonce":1978117969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6803","nonce":1978117969}]},"public_addr":"10.20.193.115:6801/1978117969","cluster_addr":"10.20.193.115:6802/1978117969","heartbeat_back_addr":"10.20.193.115:6804/1978117969","heartbeat_front_addr":"10.20.193.115:6803/1978117969","state":["exists","up"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":25,"up_from":29,"up_thru":44,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6813","nonce":1033269798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6814","nonce":1033269798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6816","nonce":1033269798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6815","nonce":1033269798}]},"public_addr":"10.20.193.115:6813/1033269798","cluster_addr":"10.20.193.115:6814/1033269798","heartbeat_back_addr":"10.20.193.115:6816/1033269798","heartbeat_front_addr":"10.20.193.115:6815/1033269798","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:32.314887+0000","dead_epoch":36},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:31.613759+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:31.523737+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:32.017476+0000","dead_epoch":26}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:31:58.889 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json osd map unique_pool_0 test 2026-02-22T23:31:59.039 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:31:59.051 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":45,"pool":"unique_pool_0","pool_id":2,"objname":"test","raw_pgid":"2.40e8aab5","pgid":"2.0","up":[3,1,2],"up_primary":3,"acting":[3,1,2],"acting_primary":3} 2026-02-22T23:31:59.051 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-02-22T23:31:59.052 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T23:31:59.898 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2026-02-22T23:32:05.153 INFO:tasks.ceph.osd.3:Stopped 2026-02-22T23:32:05.153 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 --op list |grep '"oid":"test"') sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 "$object" rm-attr hinfo_key 2026-02-22T23:32:05.154 DEBUG:teuthology.orchestra.run.trial115:> bash -e -x -c 'object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 --op list |grep '"'"'"oid":"test"'"'"') 2026-02-22T23:32:05.154 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 "$object" rm-attr hinfo_key' 2026-02-22T23:32:05.159 INFO:teuthology.orchestra.run.trial115.stderr:+ '[' -z '' ']' 2026-02-22T23:32:05.159 INFO:teuthology.orchestra.run.trial115.stderr:+ return 2026-02-22T23:32:05.159 INFO:teuthology.orchestra.run.trial115.stderr:+ case $- in 2026-02-22T23:32:05.159 INFO:teuthology.orchestra.run.trial115.stderr:+ return 2026-02-22T23:32:05.160 INFO:teuthology.orchestra.run.trial115.stderr:++ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 --op list 2026-02-22T23:32:05.160 INFO:teuthology.orchestra.run.trial115.stderr:++ grep '"oid":"test"' 2026-02-22T23:32:05.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2026-02-22T23:32:07.055 INFO:teuthology.orchestra.run.trial115.stderr:+ object='["2.0s0",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":0,"max":0}]' 2026-02-22T23:32:07.056 INFO:teuthology.orchestra.run.trial115.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 '["2.0s0",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":0,"max":0}]' rm-attr hinfo_key 2026-02-22T23:32:09.595 INFO:tasks.ceph.osd.3:Restarting daemon 2026-02-22T23:32:09.596 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2026-02-22T23:32:09.637 INFO:tasks.ceph.osd.3:Started 2026-02-22T23:32:09.638 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2026-02-22T23:32:09.698 INFO:teuthology.orchestra.run.trial115.stderr:no valid command found; 10 closest matches: 2026-02-22T23:32:09.698 INFO:teuthology.orchestra.run.trial115.stderr:0 2026-02-22T23:32:09.698 INFO:teuthology.orchestra.run.trial115.stderr:1 2026-02-22T23:32:09.698 INFO:teuthology.orchestra.run.trial115.stderr:2 2026-02-22T23:32:09.698 INFO:teuthology.orchestra.run.trial115.stderr:abort 2026-02-22T23:32:09.698 INFO:teuthology.orchestra.run.trial115.stderr:assert 2026-02-22T23:32:09.698 INFO:teuthology.orchestra.run.trial115.stderr:config diff 2026-02-22T23:32:09.699 INFO:teuthology.orchestra.run.trial115.stderr:config diff get 2026-02-22T23:32:09.699 INFO:teuthology.orchestra.run.trial115.stderr:config get 2026-02-22T23:32:09.699 INFO:teuthology.orchestra.run.trial115.stderr:config help [] 2026-02-22T23:32:09.699 INFO:teuthology.orchestra.run.trial115.stderr:config set ... 2026-02-22T23:32:09.699 INFO:teuthology.orchestra.run.trial115.stderr:admin_socket: invalid command 2026-02-22T23:32:09.700 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-22T23:32:09.701 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2026-02-22T23:32:11.161 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:32:11.160+0000 7fa99b88e8c0 -1 Falling back to public interface 2026-02-22T23:32:14.702 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2026-02-22T23:32:14.772 INFO:teuthology.orchestra.run.trial115.stderr:no valid command found; 10 closest matches: 2026-02-22T23:32:14.772 INFO:teuthology.orchestra.run.trial115.stderr:0 2026-02-22T23:32:14.772 INFO:teuthology.orchestra.run.trial115.stderr:1 2026-02-22T23:32:14.772 INFO:teuthology.orchestra.run.trial115.stderr:2 2026-02-22T23:32:14.772 INFO:teuthology.orchestra.run.trial115.stderr:abort 2026-02-22T23:32:14.772 INFO:teuthology.orchestra.run.trial115.stderr:assert 2026-02-22T23:32:14.772 INFO:teuthology.orchestra.run.trial115.stderr:config diff 2026-02-22T23:32:14.772 INFO:teuthology.orchestra.run.trial115.stderr:config diff get 2026-02-22T23:32:14.772 INFO:teuthology.orchestra.run.trial115.stderr:config get 2026-02-22T23:32:14.773 INFO:teuthology.orchestra.run.trial115.stderr:config help [] 2026-02-22T23:32:14.773 INFO:teuthology.orchestra.run.trial115.stderr:config set ... 2026-02-22T23:32:14.773 INFO:teuthology.orchestra.run.trial115.stderr:admin_socket: invalid command 2026-02-22T23:32:14.774 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-22T23:32:14.775 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2026-02-22T23:32:15.471 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:32:15.468+0000 7fa99b88e8c0 -1 osd.3 45 log_to_monitors true 2026-02-22T23:32:15.529 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:32:15.528+0000 7fa993672640 -1 osd.3 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T23:32:19.775 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2026-02-22T23:32:19.840 INFO:teuthology.orchestra.run.trial115.stdout:{ 2026-02-22T23:32:19.841 INFO:teuthology.orchestra.run.trial115.stdout: "ops": [], 2026-02-22T23:32:19.841 INFO:teuthology.orchestra.run.trial115.stdout: "num_ops": 0 2026-02-22T23:32:19.841 INFO:teuthology.orchestra.run.trial115.stdout:} 2026-02-22T23:32:19.848 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for osd.3 to be up 2026-02-22T23:32:19.848 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T23:32:19.993 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:32:19.993 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":50,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:32:17.532486+0000","last_up_change":"2026-02-22T23:32:16.528425+0000","last_in_change":"2026-02-22T23:31:36.429779+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T23:27:34.171905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-22T23:27:43.179120+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":35,"up_from":39,"up_thru":39,"down_at":36,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6809","nonce":2954119946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6810","nonce":2954119946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6812","nonce":2954119946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6811","nonce":2954119946}]},"public_addr":"10.20.193.115:6809/2954119946","cluster_addr":"10.20.193.115:6810/2954119946","heartbeat_back_addr":"10.20.193.115:6812/2954119946","heartbeat_front_addr":"10.20.193.115:6811/2954119946","state":["exists","up"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":23,"up_thru":46,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6805","nonce":2409177103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6806","nonce":2409177103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6808","nonce":2409177103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6807","nonce":2409177103}]},"public_addr":"10.20.193.115:6805/2409177103","cluster_addr":"10.20.193.115:6806/2409177103","heartbeat_back_addr":"10.20.193.115:6808/2409177103","heartbeat_front_addr":"10.20.193.115:6807/2409177103","state":["exists","up"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6801","nonce":1978117969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6802","nonce":1978117969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6804","nonce":1978117969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6803","nonce":1978117969}]},"public_addr":"10.20.193.115:6801/1978117969","cluster_addr":"10.20.193.115:6802/1978117969","heartbeat_back_addr":"10.20.193.115:6804/1978117969","heartbeat_front_addr":"10.20.193.115:6803/1978117969","state":["exists","up"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":29,"last_clean_end":45,"up_from":49,"up_thru":49,"down_at":46,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6813","nonce":428774140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6814","nonce":428774140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6816","nonce":428774140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6815","nonce":428774140}]},"public_addr":"10.20.193.115:6813/428774140","cluster_addr":"10.20.193.115:6814/428774140","heartbeat_back_addr":"10.20.193.115:6816/428774140","heartbeat_front_addr":"10.20.193.115:6815/428774140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:32.314887+0000","dead_epoch":36},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:31.613759+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:31.523737+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:32.017476+0000","dead_epoch":46}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:32:20.005 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:osd.3 is up 2026-02-22T23:32:20.005 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T23:32:20.160 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:32:20.160 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":50,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:32:17.532486+0000","last_up_change":"2026-02-22T23:32:16.528425+0000","last_in_change":"2026-02-22T23:31:36.429779+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T23:27:34.171905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-22T23:27:43.179120+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":35,"up_from":39,"up_thru":39,"down_at":36,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6809","nonce":2954119946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6810","nonce":2954119946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6812","nonce":2954119946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6811","nonce":2954119946}]},"public_addr":"10.20.193.115:6809/2954119946","cluster_addr":"10.20.193.115:6810/2954119946","heartbeat_back_addr":"10.20.193.115:6812/2954119946","heartbeat_front_addr":"10.20.193.115:6811/2954119946","state":["exists","up"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":23,"up_thru":46,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6805","nonce":2409177103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6806","nonce":2409177103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6808","nonce":2409177103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6807","nonce":2409177103}]},"public_addr":"10.20.193.115:6805/2409177103","cluster_addr":"10.20.193.115:6806/2409177103","heartbeat_back_addr":"10.20.193.115:6808/2409177103","heartbeat_front_addr":"10.20.193.115:6807/2409177103","state":["exists","up"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6801","nonce":1978117969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6802","nonce":1978117969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6804","nonce":1978117969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6803","nonce":1978117969}]},"public_addr":"10.20.193.115:6801/1978117969","cluster_addr":"10.20.193.115:6802/1978117969","heartbeat_back_addr":"10.20.193.115:6804/1978117969","heartbeat_front_addr":"10.20.193.115:6803/1978117969","state":["exists","up"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":29,"last_clean_end":45,"up_from":49,"up_thru":49,"down_at":46,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6813","nonce":428774140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6814","nonce":428774140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6816","nonce":428774140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6815","nonce":428774140}]},"public_addr":"10.20.193.115:6813/428774140","cluster_addr":"10.20.193.115:6814/428774140","heartbeat_back_addr":"10.20.193.115:6816/428774140","heartbeat_front_addr":"10.20.193.115:6815/428774140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:32.314887+0000","dead_epoch":36},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:31.613759+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:31.523737+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:32.017476+0000","dead_epoch":46}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:32:20.174 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json osd map unique_pool_0 test 2026-02-22T23:32:20.321 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:32:20.333 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":50,"pool":"unique_pool_0","pool_id":2,"objname":"test","raw_pgid":"2.40e8aab5","pgid":"2.0","up":[3,1,2],"up_primary":3,"acting":[3,1,2],"acting_primary":3} 2026-02-22T23:32:20.333 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-22T23:32:20.334 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T23:32:20.429 INFO:tasks.ceph.osd.2:Stopped 2026-02-22T23:32:20.429 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --pgid 2.0s2 --op list |grep '"oid":"test"') sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --pgid 2.0s2 "$object" rm-attr hinfo_key 2026-02-22T23:32:20.429 DEBUG:teuthology.orchestra.run.trial115:> bash -e -x -c 'object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --pgid 2.0s2 --op list |grep '"'"'"oid":"test"'"'"') 2026-02-22T23:32:20.430 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --pgid 2.0s2 "$object" rm-attr hinfo_key' 2026-02-22T23:32:20.434 INFO:teuthology.orchestra.run.trial115.stderr:+ '[' -z '' ']' 2026-02-22T23:32:20.434 INFO:teuthology.orchestra.run.trial115.stderr:+ return 2026-02-22T23:32:20.434 INFO:teuthology.orchestra.run.trial115.stderr:+ case $- in 2026-02-22T23:32:20.434 INFO:teuthology.orchestra.run.trial115.stderr:+ return 2026-02-22T23:32:20.434 INFO:teuthology.orchestra.run.trial115.stderr:++ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --pgid 2.0s2 --op list 2026-02-22T23:32:20.435 INFO:teuthology.orchestra.run.trial115.stderr:++ grep '"oid":"test"' 2026-02-22T23:32:22.724 INFO:teuthology.orchestra.run.trial115.stderr:+ object='["2.0s2",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":2,"max":0}]' 2026-02-22T23:32:22.725 INFO:teuthology.orchestra.run.trial115.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --pgid 2.0s2 '["2.0s2",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":2,"max":0}]' rm-attr hinfo_key 2026-02-22T23:32:28.162 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-22T23:32:28.162 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-02-22T23:32:28.206 INFO:tasks.ceph.osd.2:Started 2026-02-22T23:32:28.206 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-22T23:32:28.266 INFO:teuthology.orchestra.run.trial115.stderr:no valid command found; 10 closest matches: 2026-02-22T23:32:28.267 INFO:teuthology.orchestra.run.trial115.stderr:0 2026-02-22T23:32:28.267 INFO:teuthology.orchestra.run.trial115.stderr:1 2026-02-22T23:32:28.267 INFO:teuthology.orchestra.run.trial115.stderr:2 2026-02-22T23:32:28.267 INFO:teuthology.orchestra.run.trial115.stderr:abort 2026-02-22T23:32:28.267 INFO:teuthology.orchestra.run.trial115.stderr:assert 2026-02-22T23:32:28.267 INFO:teuthology.orchestra.run.trial115.stderr:config diff 2026-02-22T23:32:28.267 INFO:teuthology.orchestra.run.trial115.stderr:config diff get 2026-02-22T23:32:28.267 INFO:teuthology.orchestra.run.trial115.stderr:config get 2026-02-22T23:32:28.267 INFO:teuthology.orchestra.run.trial115.stderr:config help [] 2026-02-22T23:32:28.267 INFO:teuthology.orchestra.run.trial115.stderr:config set ... 2026-02-22T23:32:28.268 INFO:teuthology.orchestra.run.trial115.stderr:admin_socket: invalid command 2026-02-22T23:32:28.269 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-22T23:32:28.269 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-22T23:32:29.478 INFO:tasks.ceph.osd.2.trial115.stderr:2026-02-22T23:32:29.475+0000 7fc0b09598c0 -1 Falling back to public interface 2026-02-22T23:32:33.271 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-22T23:32:33.340 INFO:teuthology.orchestra.run.trial115.stderr:no valid command found; 10 closest matches: 2026-02-22T23:32:33.340 INFO:teuthology.orchestra.run.trial115.stderr:0 2026-02-22T23:32:33.340 INFO:teuthology.orchestra.run.trial115.stderr:1 2026-02-22T23:32:33.341 INFO:teuthology.orchestra.run.trial115.stderr:2 2026-02-22T23:32:33.341 INFO:teuthology.orchestra.run.trial115.stderr:abort 2026-02-22T23:32:33.341 INFO:teuthology.orchestra.run.trial115.stderr:assert 2026-02-22T23:32:33.341 INFO:teuthology.orchestra.run.trial115.stderr:bluestore allocator dump block 2026-02-22T23:32:33.341 INFO:teuthology.orchestra.run.trial115.stderr:bluestore allocator fragmentation block 2026-02-22T23:32:33.341 INFO:teuthology.orchestra.run.trial115.stderr:bluestore allocator fragmentation histogram block [] [] 2026-02-22T23:32:33.341 INFO:teuthology.orchestra.run.trial115.stderr:bluestore allocator score block 2026-02-22T23:32:33.341 INFO:teuthology.orchestra.run.trial115.stderr:config diff 2026-02-22T23:32:33.341 INFO:teuthology.orchestra.run.trial115.stderr:admin_socket: invalid command 2026-02-22T23:32:33.342 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-22T23:32:33.343 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-22T23:32:33.574 INFO:tasks.ceph.osd.2.trial115.stderr:2026-02-22T23:32:33.571+0000 7fc0b09598c0 -1 osd.2 50 log_to_monitors true 2026-02-22T23:32:33.634 INFO:tasks.ceph.osd.2.trial115.stderr:2026-02-22T23:32:33.631+0000 7fc0a873d640 -1 osd.2 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-22T23:32:38.344 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-22T23:32:38.409 INFO:teuthology.orchestra.run.trial115.stdout:{ 2026-02-22T23:32:38.409 INFO:teuthology.orchestra.run.trial115.stdout: "ops": [], 2026-02-22T23:32:38.409 INFO:teuthology.orchestra.run.trial115.stdout: "num_ops": 0 2026-02-22T23:32:38.409 INFO:teuthology.orchestra.run.trial115.stdout:} 2026-02-22T23:32:38.416 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for osd.2 to be up 2026-02-22T23:32:38.417 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T23:32:38.563 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:32:38.564 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":55,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:32:35.635530+0000","last_up_change":"2026-02-22T23:32:34.631263+0000","last_in_change":"2026-02-22T23:31:36.429779+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":11,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T23:27:34.171905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-22T23:27:43.179120+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":35,"up_from":39,"up_thru":39,"down_at":36,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6809","nonce":2954119946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6810","nonce":2954119946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6812","nonce":2954119946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6811","nonce":2954119946}]},"public_addr":"10.20.193.115:6809/2954119946","cluster_addr":"10.20.193.115:6810/2954119946","heartbeat_back_addr":"10.20.193.115:6812/2954119946","heartbeat_front_addr":"10.20.193.115:6811/2954119946","state":["exists","up"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":23,"up_thru":46,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6805","nonce":2409177103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6806","nonce":2409177103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6808","nonce":2409177103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6807","nonce":2409177103}]},"public_addr":"10.20.193.115:6805/2409177103","cluster_addr":"10.20.193.115:6806/2409177103","heartbeat_back_addr":"10.20.193.115:6808/2409177103","heartbeat_front_addr":"10.20.193.115:6807/2409177103","state":["exists","up"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":36,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6801","nonce":237665470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6802","nonce":237665470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6804","nonce":237665470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6803","nonce":237665470}]},"public_addr":"10.20.193.115:6801/237665470","cluster_addr":"10.20.193.115:6802/237665470","heartbeat_back_addr":"10.20.193.115:6804/237665470","heartbeat_front_addr":"10.20.193.115:6803/237665470","state":["exists","up"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":29,"last_clean_end":45,"up_from":49,"up_thru":54,"down_at":46,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6813","nonce":428774140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6814","nonce":428774140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6816","nonce":428774140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6815","nonce":428774140}]},"public_addr":"10.20.193.115:6813/428774140","cluster_addr":"10.20.193.115:6814/428774140","heartbeat_back_addr":"10.20.193.115:6816/428774140","heartbeat_front_addr":"10.20.193.115:6815/428774140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:32.314887+0000","dead_epoch":36},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:31.613759+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:31.523737+0000","dead_epoch":51},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:32.017476+0000","dead_epoch":46}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:32:38.576 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:osd.2 is up 2026-02-22T23:32:38.577 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 2026-02-22T23:32:38.729 INFO:tasks.ceph.mon.a.trial115.stderr:2026-02-22T23:32:38.727+0000 7f2aeaf18640 -1 mon.a@0(leader).osd e55 definitely_dead 0 2026-02-22T23:32:39.651 INFO:teuthology.orchestra.run.trial115.stderr:marked out osd.2. 2026-02-22T23:32:39.664 DEBUG:tasks.ec_inconsistent_hinfo:waiting for pg 2.0 backfill from osd.2 to osd.0 complete 2026-02-22T23:32:44.665 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:32:44.665 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:32:44.666 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:32:44.666 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:32:44.754 INFO:teuthology.orchestra.run.trial115.stdout:210453397511 2026-02-22T23:32:44.755 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:32:44.756 INFO:teuthology.orchestra.run.trial115.stdout:231928233988 2026-02-22T23:32:44.756 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:32:44.758 INFO:teuthology.orchestra.run.trial115.stdout:167503724564 2026-02-22T23:32:44.758 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:32:44.761 INFO:teuthology.orchestra.run.trial115.stdout:98784247887 2026-02-22T23:32:44.761 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:32:44.909 INFO:teuthology.orchestra.run.trial115.stdout:210453397510 2026-02-22T23:32:44.922 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397511 got 210453397510 for osd.3 2026-02-22T23:32:44.927 INFO:teuthology.orchestra.run.trial115.stdout:231928233987 2026-02-22T23:32:44.940 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928233988 got 231928233987 for osd.2 2026-02-22T23:32:44.959 INFO:teuthology.orchestra.run.trial115.stdout:167503724563 2026-02-22T23:32:44.971 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724564 got 167503724563 for osd.0 2026-02-22T23:32:44.971 INFO:teuthology.orchestra.run.trial115.stdout:98784247886 2026-02-22T23:32:44.984 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247887 got 98784247886 for osd.1 2026-02-22T23:32:45.923 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:32:45.941 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:32:45.972 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:32:45.985 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:32:46.079 INFO:teuthology.orchestra.run.trial115.stdout:210453397510 2026-02-22T23:32:46.094 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397511 got 210453397510 for osd.3 2026-02-22T23:32:46.119 INFO:teuthology.orchestra.run.trial115.stdout:231928233987 2026-02-22T23:32:46.132 INFO:teuthology.orchestra.run.trial115.stdout:167503724563 2026-02-22T23:32:46.133 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928233988 got 231928233987 for osd.2 2026-02-22T23:32:46.146 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724564 got 167503724563 for osd.0 2026-02-22T23:32:46.166 INFO:teuthology.orchestra.run.trial115.stdout:98784247886 2026-02-22T23:32:46.178 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247887 got 98784247886 for osd.1 2026-02-22T23:32:47.096 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:32:47.134 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:32:47.147 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:32:47.179 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:32:47.255 INFO:teuthology.orchestra.run.trial115.stdout:210453397512 2026-02-22T23:32:47.268 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397511 got 210453397512 for osd.3 2026-02-22T23:32:47.268 DEBUG:teuthology.parallel:result is None 2026-02-22T23:32:47.302 INFO:teuthology.orchestra.run.trial115.stdout:231928233988 2026-02-22T23:32:47.316 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928233988 got 231928233988 for osd.2 2026-02-22T23:32:47.316 DEBUG:teuthology.parallel:result is None 2026-02-22T23:32:47.328 INFO:teuthology.orchestra.run.trial115.stdout:98784247887 2026-02-22T23:32:47.339 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247887 got 98784247887 for osd.1 2026-02-22T23:32:47.340 DEBUG:teuthology.parallel:result is None 2026-02-22T23:32:47.345 INFO:teuthology.orchestra.run.trial115.stdout:167503724564 2026-02-22T23:32:47.359 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724564 got 167503724564 for osd.0 2026-02-22T23:32:47.359 DEBUG:teuthology.parallel:result is None 2026-02-22T23:32:47.360 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:32:47.544 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:32:47.545 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:32:47.555 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2026-02-22T23:32:46.173846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":65746,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71934,"num_bytes_recovered":589587017,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":821160,"kb_used_data":558760,"kb_used_omap":14,"kb_used_meta":262321,"kb_avail":282294360,"statfs":{"total":289910292480,"available":289069424640,"internally_reserved":0,"allocated":572170240,"data_stored":571919709,"data_compressed":10360,"data_compressed_allocated":57344,"data_compressed_original":688128,"omap_allocated":14890,"internal_metadata":268617174},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":65746,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2854,"num_bytes_recovered":23379968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.521944"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":212650,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:32:45.532620+0000","last_change":"2026-02-22T23:32:41.678206+0000","last_active":"2026-02-22T23:32:45.532620+0000","last_peered":"2026-02-22T23:32:45.532620+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:32:45.532620+0000","last_undegraded":"2026-02-22T23:32:45.532620+0000","last_fullsized":"2026-02-22T23:32:45.532620+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":65746,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71932,"num_bytes_recovered":589259337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":48,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662608+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662608+0000","last_peered":"2026-02-22T23:32:41.662608+0000","last_clean":"2026-02-22T23:32:41.662608+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662608+0000","last_undegraded":"2026-02-22T23:32:41.662608+0000","last_fullsized":"2026-02-22T23:32:41.662608+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":65746,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71932,"num_bytes_recovered":589259337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":852152320,"data_stored":852176896,"data_compressed":16108,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724564,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55744,"kb_used_data":9008,"kb_used_omap":4,"kb_used_meta":46715,"kb_avail":94316096,"statfs":{"total":96636764160,"available":96579682304,"internally_reserved":0,"allocated":9224192,"data_stored":9235231,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":4401,"internal_metadata":47836879},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397512,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314448,"kb_used_data":274868,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057392,"statfs":{"total":96636764160,"available":96314769408,"internally_reserved":0,"allocated":281464832,"data_stored":281180447,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928233988,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247887,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450968,"kb_used_data":274884,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920872,"statfs":{"total":96636764160,"available":96174972928,"internally_reserved":0,"allocated":281481216,"data_stored":281504031,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9211904,"data_stored":9211904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:32:47.556 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 212650, 'reported_epoch': 58, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:32:45.532620+0000', 'last_change': '2026-02-22T23:32:41.678206+0000', 'last_active': '2026-02-22T23:32:45.532620+0000', 'last_peered': '2026-02-22T23:32:45.532620+0000', 'last_clean': '2026-02-22T23:32:35.643295+0000', 'last_became_active': '2026-02-22T23:32:41.677538+0000', 'last_became_peered': '2026-02-22T23:32:41.677538+0000', 'last_unstale': '2026-02-22T23:32:45.532620+0000', 'last_undegraded': '2026-02-22T23:32:45.532620+0000', 'last_fullsized': '2026-02-22T23:32:45.532620+0000', 'mapping_epoch': 57, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 55, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 65746, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 71932, 'num_bytes_recovered': 589259337, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:32:47.556 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:32:52.557 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:32:52.558 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:32:52.559 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:32:52.559 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:32:52.644 INFO:teuthology.orchestra.run.trial115.stdout:231928233990 2026-02-22T23:32:52.644 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:32:52.646 INFO:teuthology.orchestra.run.trial115.stdout:210453397514 2026-02-22T23:32:52.646 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:32:52.646 INFO:teuthology.orchestra.run.trial115.stdout:98784247889 2026-02-22T23:32:52.646 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:32:52.647 INFO:teuthology.orchestra.run.trial115.stdout:167503724567 2026-02-22T23:32:52.647 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:32:52.797 INFO:teuthology.orchestra.run.trial115.stdout:231928233989 2026-02-22T23:32:52.801 INFO:teuthology.orchestra.run.trial115.stdout:98784247888 2026-02-22T23:32:52.805 INFO:teuthology.orchestra.run.trial115.stdout:167503724566 2026-02-22T23:32:52.811 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928233990 got 231928233989 for osd.2 2026-02-22T23:32:52.813 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247889 got 98784247888 for osd.1 2026-02-22T23:32:52.815 INFO:teuthology.orchestra.run.trial115.stdout:210453397513 2026-02-22T23:32:52.819 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724567 got 167503724566 for osd.0 2026-02-22T23:32:52.829 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397514 got 210453397513 for osd.3 2026-02-22T23:32:53.811 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:32:53.814 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:32:53.820 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:32:53.830 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:32:53.963 INFO:teuthology.orchestra.run.trial115.stdout:231928233989 2026-02-22T23:32:53.977 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928233990 got 231928233989 for osd.2 2026-02-22T23:32:54.013 INFO:teuthology.orchestra.run.trial115.stdout:210453397513 2026-02-22T23:32:54.025 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397514 got 210453397513 for osd.3 2026-02-22T23:32:54.029 INFO:teuthology.orchestra.run.trial115.stdout:98784247888 2026-02-22T23:32:54.037 INFO:teuthology.orchestra.run.trial115.stdout:167503724566 2026-02-22T23:32:54.041 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247889 got 98784247888 for osd.1 2026-02-22T23:32:54.052 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724567 got 167503724566 for osd.0 2026-02-22T23:32:54.978 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:32:55.026 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:32:55.042 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:32:55.052 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:32:55.135 INFO:teuthology.orchestra.run.trial115.stdout:231928233991 2026-02-22T23:32:55.148 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928233990 got 231928233991 for osd.2 2026-02-22T23:32:55.148 DEBUG:teuthology.parallel:result is None 2026-02-22T23:32:55.178 INFO:teuthology.orchestra.run.trial115.stdout:210453397514 2026-02-22T23:32:55.191 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397514 got 210453397514 for osd.3 2026-02-22T23:32:55.191 DEBUG:teuthology.parallel:result is None 2026-02-22T23:32:55.220 INFO:teuthology.orchestra.run.trial115.stdout:167503724567 2026-02-22T23:32:55.229 INFO:teuthology.orchestra.run.trial115.stdout:98784247890 2026-02-22T23:32:55.231 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724567 got 167503724567 for osd.0 2026-02-22T23:32:55.232 DEBUG:teuthology.parallel:result is None 2026-02-22T23:32:55.244 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247889 got 98784247890 for osd.1 2026-02-22T23:32:55.245 DEBUG:teuthology.parallel:result is None 2026-02-22T23:32:55.245 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:32:55.434 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:32:55.435 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:32:55.445 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2026-02-22T23:32:54.174835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":60173,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77507,"num_bytes_recovered":635241033,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":886572,"kb_used_data":583916,"kb_used_omap":14,"kb_used_meta":302577,"kb_avail":282228948,"statfs":{"total":289910292480,"available":289002442752,"internally_reserved":0,"allocated":597929984,"data_stored":597679453,"data_compressed":10360,"data_compressed_allocated":57344,"data_compressed_original":688128,"omap_allocated":14890,"internal_metadata":309839318},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":60173,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8427,"num_bytes_recovered":69033984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001434"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":223796,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:32:52.634601+0000","last_change":"2026-02-22T23:32:41.678206+0000","last_active":"2026-02-22T23:32:52.634601+0000","last_peered":"2026-02-22T23:32:52.634601+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:32:52.634601+0000","last_undegraded":"2026-02-22T23:32:52.634601+0000","last_fullsized":"2026-02-22T23:32:52.634601+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":60173,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77505,"num_bytes_recovered":634913353,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":48,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662608+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662608+0000","last_peered":"2026-02-22T23:32:41.662608+0000","last_clean":"2026-02-22T23:32:41.662608+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662608+0000","last_undegraded":"2026-02-22T23:32:41.662608+0000","last_fullsized":"2026-02-22T23:32:41.662608+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":60173,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77505,"num_bytes_recovered":634913353,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":877469696,"data_stored":877494272,"data_compressed":16108,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121156,"kb_used_data":34164,"kb_used_omap":4,"kb_used_meta":86971,"kb_avail":94250684,"statfs":{"total":96636764160,"available":96512700416,"internally_reserved":0,"allocated":34983936,"data_stored":34994975,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":4401,"internal_metadata":89059023},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397514,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314448,"kb_used_data":274868,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057392,"statfs":{"total":96636764160,"available":96314769408,"internally_reserved":0,"allocated":281464832,"data_stored":281180447,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928233991,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247890,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450968,"kb_used_data":274884,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920872,"statfs":{"total":96636764160,"available":96174972928,"internally_reserved":0,"allocated":281481216,"data_stored":281504031,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34529280,"data_stored":34529280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:32:55.446 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 223796, 'reported_epoch': 58, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:32:52.634601+0000', 'last_change': '2026-02-22T23:32:41.678206+0000', 'last_active': '2026-02-22T23:32:52.634601+0000', 'last_peered': '2026-02-22T23:32:52.634601+0000', 'last_clean': '2026-02-22T23:32:35.643295+0000', 'last_became_active': '2026-02-22T23:32:41.677538+0000', 'last_became_peered': '2026-02-22T23:32:41.677538+0000', 'last_unstale': '2026-02-22T23:32:52.634601+0000', 'last_undegraded': '2026-02-22T23:32:52.634601+0000', 'last_fullsized': '2026-02-22T23:32:52.634601+0000', 'mapping_epoch': 57, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 55, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 60173, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 77505, 'num_bytes_recovered': 634913353, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:32:55.446 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:33:00.448 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:33:00.449 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:33:00.449 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:33:00.450 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:33:00.537 INFO:teuthology.orchestra.run.trial115.stdout:167503724569 2026-02-22T23:33:00.537 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:00.540 INFO:teuthology.orchestra.run.trial115.stdout:231928233993 2026-02-22T23:33:00.540 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:00.542 INFO:teuthology.orchestra.run.trial115.stdout:210453397516 2026-02-22T23:33:00.543 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:00.545 INFO:teuthology.orchestra.run.trial115.stdout:98784247892 2026-02-22T23:33:00.546 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:00.712 INFO:teuthology.orchestra.run.trial115.stdout:167503724568 2026-02-22T23:33:00.717 INFO:teuthology.orchestra.run.trial115.stdout:231928233992 2026-02-22T23:33:00.725 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724569 got 167503724568 for osd.0 2026-02-22T23:33:00.731 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928233993 got 231928233992 for osd.2 2026-02-22T23:33:00.740 INFO:teuthology.orchestra.run.trial115.stdout:98784247891 2026-02-22T23:33:00.740 INFO:teuthology.orchestra.run.trial115.stdout:210453397515 2026-02-22T23:33:00.753 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397516 got 210453397515 for osd.3 2026-02-22T23:33:00.754 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247892 got 98784247891 for osd.1 2026-02-22T23:33:01.726 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:01.731 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:01.754 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:01.755 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:01.889 INFO:teuthology.orchestra.run.trial115.stdout:167503724568 2026-02-22T23:33:01.904 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724569 got 167503724568 for osd.0 2026-02-22T23:33:01.927 INFO:teuthology.orchestra.run.trial115.stdout:98784247891 2026-02-22T23:33:01.927 INFO:teuthology.orchestra.run.trial115.stdout:210453397515 2026-02-22T23:33:01.939 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247892 got 98784247891 for osd.1 2026-02-22T23:33:01.940 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397516 got 210453397515 for osd.3 2026-02-22T23:33:01.941 INFO:teuthology.orchestra.run.trial115.stdout:231928233992 2026-02-22T23:33:01.954 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928233993 got 231928233992 for osd.2 2026-02-22T23:33:02.904 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:02.940 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:02.941 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:02.955 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:03.050 INFO:teuthology.orchestra.run.trial115.stdout:167503724570 2026-02-22T23:33:03.063 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724569 got 167503724570 for osd.0 2026-02-22T23:33:03.063 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:03.100 INFO:teuthology.orchestra.run.trial115.stdout:210453397517 2026-02-22T23:33:03.109 INFO:teuthology.orchestra.run.trial115.stdout:98784247892 2026-02-22T23:33:03.112 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397516 got 210453397517 for osd.3 2026-02-22T23:33:03.113 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:03.124 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247892 got 98784247892 for osd.1 2026-02-22T23:33:03.124 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:03.144 INFO:teuthology.orchestra.run.trial115.stdout:231928233993 2026-02-22T23:33:03.156 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928233993 got 231928233993 for osd.2 2026-02-22T23:33:03.156 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:03.157 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:33:03.341 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:33:03.341 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:33:03.352 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2026-02-22T23:33:02.175909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":53988,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83692,"num_bytes_recovered":685908740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":947536,"kb_used_data":608784,"kb_used_omap":14,"kb_used_meta":338673,"kb_avail":282167984,"statfs":{"total":289910292480,"available":288940015616,"internally_reserved":0,"allocated":623394816,"data_stored":623148381,"data_compressed":13827,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14890,"internal_metadata":346801622},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-11758,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11758,"num_bytes_recovered":96321723,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001563"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":236166,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:33:00.533667+0000","last_change":"2026-02-22T23:32:41.678206+0000","last_active":"2026-02-22T23:33:00.533667+0000","last_peered":"2026-02-22T23:33:00.533667+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:33:00.533667+0000","last_undegraded":"2026-02-22T23:33:00.533667+0000","last_fullsized":"2026-02-22T23:33:00.533667+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":53988,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83690,"num_bytes_recovered":685581060,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":48,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662608+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662608+0000","last_peered":"2026-02-22T23:32:41.662608+0000","last_clean":"2026-02-22T23:32:41.662608+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662608+0000","last_undegraded":"2026-02-22T23:32:41.662608+0000","last_fullsized":"2026-02-22T23:32:41.662608+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":53988,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83690,"num_bytes_recovered":685581060,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":904945664,"data_stored":904974336,"data_compressed":19575,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182120,"kb_used_data":59032,"kb_used_omap":4,"kb_used_meta":123067,"kb_avail":94189720,"statfs":{"total":96636764160,"available":96450273280,"internally_reserved":0,"allocated":60448768,"data_stored":60463903,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":126021327},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314448,"kb_used_data":274868,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057392,"statfs":{"total":96636764160,"available":96314769408,"internally_reserved":0,"allocated":281464832,"data_stored":281180447,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928233993,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247892,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450968,"kb_used_data":274884,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920872,"statfs":{"total":96636764160,"available":96174972928,"internally_reserved":0,"allocated":281481216,"data_stored":281504031,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62005248,"data_stored":62009344,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:33:03.353 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 236166, 'reported_epoch': 58, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:33:00.533667+0000', 'last_change': '2026-02-22T23:32:41.678206+0000', 'last_active': '2026-02-22T23:33:00.533667+0000', 'last_peered': '2026-02-22T23:33:00.533667+0000', 'last_clean': '2026-02-22T23:32:35.643295+0000', 'last_became_active': '2026-02-22T23:32:41.677538+0000', 'last_became_peered': '2026-02-22T23:32:41.677538+0000', 'last_unstale': '2026-02-22T23:33:00.533667+0000', 'last_undegraded': '2026-02-22T23:33:00.533667+0000', 'last_fullsized': '2026-02-22T23:33:00.533667+0000', 'mapping_epoch': 57, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 55, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 53988, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83690, 'num_bytes_recovered': 685581060, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:33:03.353 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:33:08.355 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:33:08.356 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:33:08.356 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:33:08.357 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:33:08.446 INFO:teuthology.orchestra.run.trial115.stdout:231928233995 2026-02-22T23:33:08.447 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:08.449 INFO:teuthology.orchestra.run.trial115.stdout:167503724572 2026-02-22T23:33:08.450 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:08.451 INFO:teuthology.orchestra.run.trial115.stdout:98784247895 2026-02-22T23:33:08.451 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:08.453 INFO:teuthology.orchestra.run.trial115.stdout:210453397519 2026-02-22T23:33:08.453 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:08.603 INFO:teuthology.orchestra.run.trial115.stdout:231928233994 2026-02-22T23:33:08.617 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928233995 got 231928233994 for osd.2 2026-02-22T23:33:08.646 INFO:teuthology.orchestra.run.trial115.stdout:210453397518 2026-02-22T23:33:08.650 INFO:teuthology.orchestra.run.trial115.stdout:167503724571 2026-02-22T23:33:08.658 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397519 got 210453397518 for osd.3 2026-02-22T23:33:08.662 INFO:teuthology.orchestra.run.trial115.stdout:98784247893 2026-02-22T23:33:08.663 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724572 got 167503724571 for osd.0 2026-02-22T23:33:08.675 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247895 got 98784247893 for osd.1 2026-02-22T23:33:09.618 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:09.659 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:09.664 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:09.676 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:09.779 INFO:teuthology.orchestra.run.trial115.stdout:231928233994 2026-02-22T23:33:09.793 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928233995 got 231928233994 for osd.2 2026-02-22T23:33:09.824 INFO:teuthology.orchestra.run.trial115.stdout:210453397518 2026-02-22T23:33:09.838 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397519 got 210453397518 for osd.3 2026-02-22T23:33:09.867 INFO:teuthology.orchestra.run.trial115.stdout:98784247893 2026-02-22T23:33:09.879 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247895 got 98784247893 for osd.1 2026-02-22T23:33:09.895 INFO:teuthology.orchestra.run.trial115.stdout:167503724571 2026-02-22T23:33:09.911 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724572 got 167503724571 for osd.0 2026-02-22T23:33:10.794 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:10.838 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:10.880 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:10.912 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:10.939 INFO:teuthology.orchestra.run.trial115.stdout:231928233996 2026-02-22T23:33:10.951 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928233995 got 231928233996 for osd.2 2026-02-22T23:33:10.951 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:10.990 INFO:teuthology.orchestra.run.trial115.stdout:210453397519 2026-02-22T23:33:11.004 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397519 got 210453397519 for osd.3 2026-02-22T23:33:11.004 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:11.059 INFO:teuthology.orchestra.run.trial115.stdout:98784247895 2026-02-22T23:33:11.073 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247895 got 98784247895 for osd.1 2026-02-22T23:33:11.074 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:11.082 INFO:teuthology.orchestra.run.trial115.stdout:167503724572 2026-02-22T23:33:11.094 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724572 got 167503724572 for osd.0 2026-02-22T23:33:11.094 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:11.094 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:33:11.276 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:33:11.276 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:33:11.287 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2026-02-22T23:33:10.176956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":47792,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89888,"num_bytes_recovered":736666372,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":965804,"kb_used_data":631212,"kb_used_omap":14,"kb_used_meta":334513,"kb_avail":282149716,"statfs":{"total":289910292480,"available":288921309184,"internally_reserved":0,"allocated":646361088,"data_stored":646114653,"data_compressed":13827,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14890,"internal_metadata":342541782},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-10099,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10099,"num_bytes_recovered":82731195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001583"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":248558,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:33:08.441250+0000","last_change":"2026-02-22T23:32:41.678206+0000","last_active":"2026-02-22T23:33:08.441250+0000","last_peered":"2026-02-22T23:33:08.441250+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:33:08.441250+0000","last_undegraded":"2026-02-22T23:33:08.441250+0000","last_fullsized":"2026-02-22T23:33:08.441250+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":47792,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89886,"num_bytes_recovered":736338692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":48,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662608+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662608+0000","last_peered":"2026-02-22T23:32:41.662608+0000","last_clean":"2026-02-22T23:32:41.662608+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662608+0000","last_undegraded":"2026-02-22T23:32:41.662608+0000","last_fullsized":"2026-02-22T23:32:41.662608+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":47792,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89886,"num_bytes_recovered":736338692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":928161792,"data_stored":928190464,"data_compressed":19575,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724572,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":200388,"kb_used_data":81460,"kb_used_omap":4,"kb_used_meta":118907,"kb_avail":94171452,"statfs":{"total":96636764160,"available":96431566848,"internally_reserved":0,"allocated":83415040,"data_stored":83430175,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":121761487},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397519,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314448,"kb_used_data":274868,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057392,"statfs":{"total":96636764160,"available":96314769408,"internally_reserved":0,"allocated":281464832,"data_stored":281180447,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928233996,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247895,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450968,"kb_used_data":274884,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920872,"statfs":{"total":96636764160,"available":96174972928,"internally_reserved":0,"allocated":281481216,"data_stored":281504031,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":85221376,"data_stored":85225472,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:33:11.289 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 248558, 'reported_epoch': 58, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:33:08.441250+0000', 'last_change': '2026-02-22T23:32:41.678206+0000', 'last_active': '2026-02-22T23:33:08.441250+0000', 'last_peered': '2026-02-22T23:33:08.441250+0000', 'last_clean': '2026-02-22T23:32:35.643295+0000', 'last_became_active': '2026-02-22T23:32:41.677538+0000', 'last_became_peered': '2026-02-22T23:32:41.677538+0000', 'last_unstale': '2026-02-22T23:33:08.441250+0000', 'last_undegraded': '2026-02-22T23:33:08.441250+0000', 'last_fullsized': '2026-02-22T23:33:08.441250+0000', 'mapping_epoch': 57, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 55, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 47792, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 89886, 'num_bytes_recovered': 736338692, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:33:11.289 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:33:16.290 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:33:16.291 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:33:16.292 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:33:16.292 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:33:16.381 INFO:teuthology.orchestra.run.trial115.stdout:210453397522 2026-02-22T23:33:16.381 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:16.383 INFO:teuthology.orchestra.run.trial115.stdout:98784247897 2026-02-22T23:33:16.383 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:16.384 INFO:teuthology.orchestra.run.trial115.stdout:167503724575 2026-02-22T23:33:16.384 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:16.386 INFO:teuthology.orchestra.run.trial115.stdout:231928233998 2026-02-22T23:33:16.387 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:16.530 INFO:teuthology.orchestra.run.trial115.stdout:210453397521 2026-02-22T23:33:16.543 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397522 got 210453397521 for osd.3 2026-02-22T23:33:16.575 INFO:teuthology.orchestra.run.trial115.stdout:98784247896 2026-02-22T23:33:16.581 INFO:teuthology.orchestra.run.trial115.stdout:167503724573 2026-02-22T23:33:16.586 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247897 got 98784247896 for osd.1 2026-02-22T23:33:16.587 INFO:teuthology.orchestra.run.trial115.stdout:231928233997 2026-02-22T23:33:16.593 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724575 got 167503724573 for osd.0 2026-02-22T23:33:16.600 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928233998 got 231928233997 for osd.2 2026-02-22T23:33:17.544 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:17.588 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:17.594 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:17.601 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:17.691 INFO:teuthology.orchestra.run.trial115.stdout:210453397521 2026-02-22T23:33:17.704 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397522 got 210453397521 for osd.3 2026-02-22T23:33:17.735 INFO:teuthology.orchestra.run.trial115.stdout:98784247896 2026-02-22T23:33:17.747 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247897 got 98784247896 for osd.1 2026-02-22T23:33:17.781 INFO:teuthology.orchestra.run.trial115.stdout:231928233997 2026-02-22T23:33:17.792 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928233998 got 231928233997 for osd.2 2026-02-22T23:33:17.795 INFO:teuthology.orchestra.run.trial115.stdout:167503724573 2026-02-22T23:33:17.809 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724575 got 167503724573 for osd.0 2026-02-22T23:33:18.705 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:18.748 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:18.793 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:18.809 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:18.851 INFO:teuthology.orchestra.run.trial115.stdout:210453397522 2026-02-22T23:33:18.865 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397522 got 210453397522 for osd.3 2026-02-22T23:33:18.865 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:18.895 INFO:teuthology.orchestra.run.trial115.stdout:98784247897 2026-02-22T23:33:18.910 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247897 got 98784247897 for osd.1 2026-02-22T23:33:18.910 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:18.938 INFO:teuthology.orchestra.run.trial115.stdout:231928233998 2026-02-22T23:33:18.951 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928233998 got 231928233998 for osd.2 2026-02-22T23:33:18.951 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:18.996 INFO:teuthology.orchestra.run.trial115.stdout:167503724575 2026-02-22T23:33:19.009 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724575 got 167503724575 for osd.0 2026-02-22T23:33:19.009 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:19.009 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:33:19.155 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:33:19.155 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:33:19.166 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2026-02-22T23:33:18.177974+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":41645,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96035,"num_bytes_recovered":787022596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1026688,"kb_used_data":656000,"kb_used_omap":14,"kb_used_meta":370609,"kb_avail":282088832,"statfs":{"total":289910292480,"available":288858963968,"internally_reserved":0,"allocated":671744000,"data_stored":671497565,"data_compressed":13827,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14890,"internal_metadata":379504086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-8366,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8366,"num_bytes_recovered":68534272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001466"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":260852,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:33:16.369887+0000","last_change":"2026-02-22T23:32:41.678206+0000","last_active":"2026-02-22T23:33:16.369887+0000","last_peered":"2026-02-22T23:33:16.369887+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:33:16.369887+0000","last_undegraded":"2026-02-22T23:33:16.369887+0000","last_fullsized":"2026-02-22T23:33:16.369887+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":41645,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96033,"num_bytes_recovered":786694916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":48,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662608+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662608+0000","last_peered":"2026-02-22T23:32:41.662608+0000","last_clean":"2026-02-22T23:32:41.662608+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662608+0000","last_undegraded":"2026-02-22T23:32:41.662608+0000","last_fullsized":"2026-02-22T23:32:41.662608+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":41645,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96033,"num_bytes_recovered":786694916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":953352192,"data_stored":953380864,"data_compressed":19575,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724575,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":261272,"kb_used_data":106248,"kb_used_omap":4,"kb_used_meta":155003,"kb_avail":94110568,"statfs":{"total":96636764160,"available":96369221632,"internally_reserved":0,"allocated":108797952,"data_stored":108813087,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":158723791},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397522,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314448,"kb_used_data":274868,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057392,"statfs":{"total":96636764160,"available":96314769408,"internally_reserved":0,"allocated":281464832,"data_stored":281180447,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928233998,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247897,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450968,"kb_used_data":274884,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920872,"statfs":{"total":96636764160,"available":96174972928,"internally_reserved":0,"allocated":281481216,"data_stored":281504031,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":110411776,"data_stored":110415872,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:33:19.167 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 260852, 'reported_epoch': 58, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:33:16.369887+0000', 'last_change': '2026-02-22T23:32:41.678206+0000', 'last_active': '2026-02-22T23:33:16.369887+0000', 'last_peered': '2026-02-22T23:33:16.369887+0000', 'last_clean': '2026-02-22T23:32:35.643295+0000', 'last_became_active': '2026-02-22T23:32:41.677538+0000', 'last_became_peered': '2026-02-22T23:32:41.677538+0000', 'last_unstale': '2026-02-22T23:33:16.369887+0000', 'last_undegraded': '2026-02-22T23:33:16.369887+0000', 'last_fullsized': '2026-02-22T23:33:16.369887+0000', 'mapping_epoch': 57, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 55, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 41645, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 96033, 'num_bytes_recovered': 786694916, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:33:19.167 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:33:24.169 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:33:24.170 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:33:24.170 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:33:24.170 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:33:24.261 INFO:teuthology.orchestra.run.trial115.stdout:210453397524 2026-02-22T23:33:24.261 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:24.264 INFO:teuthology.orchestra.run.trial115.stdout:98784247900 2026-02-22T23:33:24.265 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:24.266 INFO:teuthology.orchestra.run.trial115.stdout:231928234001 2026-02-22T23:33:24.267 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:24.269 INFO:teuthology.orchestra.run.trial115.stdout:167503724577 2026-02-22T23:33:24.270 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:24.425 INFO:teuthology.orchestra.run.trial115.stdout:98784247899 2026-02-22T23:33:24.437 INFO:teuthology.orchestra.run.trial115.stdout:210453397523 2026-02-22T23:33:24.439 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247900 got 98784247899 for osd.1 2026-02-22T23:33:24.451 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397524 got 210453397523 for osd.3 2026-02-22T23:33:24.462 INFO:teuthology.orchestra.run.trial115.stdout:167503724576 2026-02-22T23:33:24.473 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724577 got 167503724576 for osd.0 2026-02-22T23:33:24.477 INFO:teuthology.orchestra.run.trial115.stdout:231928234000 2026-02-22T23:33:24.490 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928234001 got 231928234000 for osd.2 2026-02-22T23:33:25.440 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:25.452 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:25.474 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:25.491 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:25.588 INFO:teuthology.orchestra.run.trial115.stdout:98784247899 2026-02-22T23:33:25.601 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247900 got 98784247899 for osd.1 2026-02-22T23:33:25.647 INFO:teuthology.orchestra.run.trial115.stdout:167503724576 2026-02-22T23:33:25.650 INFO:teuthology.orchestra.run.trial115.stdout:210453397523 2026-02-22T23:33:25.661 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724577 got 167503724576 for osd.0 2026-02-22T23:33:25.665 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397524 got 210453397523 for osd.3 2026-02-22T23:33:25.672 INFO:teuthology.orchestra.run.trial115.stdout:231928234000 2026-02-22T23:33:25.684 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928234001 got 231928234000 for osd.2 2026-02-22T23:33:26.602 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:26.662 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:26.666 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:26.685 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:26.748 INFO:teuthology.orchestra.run.trial115.stdout:98784247900 2026-02-22T23:33:26.761 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247900 got 98784247900 for osd.1 2026-02-22T23:33:26.762 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:26.818 INFO:teuthology.orchestra.run.trial115.stdout:167503724577 2026-02-22T23:33:26.830 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724577 got 167503724577 for osd.0 2026-02-22T23:33:26.831 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:26.853 INFO:teuthology.orchestra.run.trial115.stdout:210453397525 2026-02-22T23:33:26.858 INFO:teuthology.orchestra.run.trial115.stdout:231928234001 2026-02-22T23:33:26.866 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397524 got 210453397525 for osd.3 2026-02-22T23:33:26.866 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:26.870 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928234001 got 231928234001 for osd.2 2026-02-22T23:33:26.871 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:26.871 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:33:27.070 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:33:27.071 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:33:27.084 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2026-02-22T23:33:26.179013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34383,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103297,"num_bytes_recovered":846512900,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1092384,"kb_used_data":681440,"kb_used_omap":14,"kb_used_meta":410865,"kb_avail":282023136,"statfs":{"total":289910292480,"available":288791691264,"internally_reserved":0,"allocated":697794560,"data_stored":697548125,"data_compressed":13827,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14890,"internal_metadata":420726230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-11758,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11758,"num_bytes_recovered":96321536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001562"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":275376,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:33:25.535518+0000","last_change":"2026-02-22T23:32:41.678206+0000","last_active":"2026-02-22T23:33:25.535518+0000","last_peered":"2026-02-22T23:33:25.535518+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:33:25.535518+0000","last_undegraded":"2026-02-22T23:33:25.535518+0000","last_fullsized":"2026-02-22T23:33:25.535518+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34383,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103295,"num_bytes_recovered":846185220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":48,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662608+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662608+0000","last_peered":"2026-02-22T23:32:41.662608+0000","last_clean":"2026-02-22T23:32:41.662608+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662608+0000","last_undegraded":"2026-02-22T23:32:41.662608+0000","last_fullsized":"2026-02-22T23:32:41.662608+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34383,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103295,"num_bytes_recovered":846185220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":979001344,"data_stored":979030016,"data_compressed":19575,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724577,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326968,"kb_used_data":131688,"kb_used_omap":4,"kb_used_meta":195259,"kb_avail":94044872,"statfs":{"total":96636764160,"available":96301948928,"internally_reserved":0,"allocated":134848512,"data_stored":134863647,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":199945935},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397525,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314448,"kb_used_data":274868,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057392,"statfs":{"total":96636764160,"available":96314769408,"internally_reserved":0,"allocated":281464832,"data_stored":281180447,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234001,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247900,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450968,"kb_used_data":274884,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920872,"statfs":{"total":96636764160,"available":96174972928,"internally_reserved":0,"allocated":281481216,"data_stored":281504031,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":136060928,"data_stored":136065024,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:33:27.085 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 275376, 'reported_epoch': 58, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:33:25.535518+0000', 'last_change': '2026-02-22T23:32:41.678206+0000', 'last_active': '2026-02-22T23:33:25.535518+0000', 'last_peered': '2026-02-22T23:33:25.535518+0000', 'last_clean': '2026-02-22T23:32:35.643295+0000', 'last_became_active': '2026-02-22T23:32:41.677538+0000', 'last_became_peered': '2026-02-22T23:32:41.677538+0000', 'last_unstale': '2026-02-22T23:33:25.535518+0000', 'last_undegraded': '2026-02-22T23:33:25.535518+0000', 'last_fullsized': '2026-02-22T23:33:25.535518+0000', 'mapping_epoch': 57, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 55, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 34383, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 103295, 'num_bytes_recovered': 846185220, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:33:27.085 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:33:32.086 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:33:32.087 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:33:32.088 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:33:32.088 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:33:32.178 INFO:teuthology.orchestra.run.trial115.stdout:210453397527 2026-02-22T23:33:32.179 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:32.180 INFO:teuthology.orchestra.run.trial115.stdout:98784247902 2026-02-22T23:33:32.180 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:32.186 INFO:teuthology.orchestra.run.trial115.stdout:167503724580 2026-02-22T23:33:32.186 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:32.187 INFO:teuthology.orchestra.run.trial115.stdout:231928234003 2026-02-22T23:33:32.188 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:32.382 INFO:teuthology.orchestra.run.trial115.stdout:210453397527 2026-02-22T23:33:32.384 INFO:teuthology.orchestra.run.trial115.stdout:167503724580 2026-02-22T23:33:32.385 INFO:teuthology.orchestra.run.trial115.stdout:98784247902 2026-02-22T23:33:32.395 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397527 got 210453397527 for osd.3 2026-02-22T23:33:32.395 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:32.398 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724580 got 167503724580 for osd.0 2026-02-22T23:33:32.399 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:32.399 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247902 got 98784247902 for osd.1 2026-02-22T23:33:32.399 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:32.423 INFO:teuthology.orchestra.run.trial115.stdout:231928234003 2026-02-22T23:33:32.436 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928234003 got 231928234003 for osd.2 2026-02-22T23:33:32.436 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:32.436 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:33:32.634 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:33:32.634 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:33:32.647 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2026-02-22T23:33:32.179667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":29110,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108570,"num_bytes_recovered":889709316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1154192,"kb_used_data":707152,"kb_used_omap":14,"kb_used_meta":446961,"kb_avail":281961328,"statfs":{"total":289910292480,"available":288728399872,"internally_reserved":0,"allocated":724123648,"data_stored":723877213,"data_compressed":13827,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14890,"internal_metadata":457688534},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-12535,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12535,"num_bytes_recovered":102686720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001441"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":285922,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:33:32.166354+0000","last_change":"2026-02-22T23:32:41.678206+0000","last_active":"2026-02-22T23:33:32.166354+0000","last_peered":"2026-02-22T23:33:32.166354+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:33:32.166354+0000","last_undegraded":"2026-02-22T23:33:32.166354+0000","last_fullsized":"2026-02-22T23:33:32.166354+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":29110,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108568,"num_bytes_recovered":889381636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":48,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662608+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662608+0000","last_peered":"2026-02-22T23:32:41.662608+0000","last_clean":"2026-02-22T23:32:41.662608+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662608+0000","last_undegraded":"2026-02-22T23:32:41.662608+0000","last_fullsized":"2026-02-22T23:32:41.662608+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":29110,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108568,"num_bytes_recovered":889381636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1004707840,"data_stored":1004736512,"data_compressed":19575,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724580,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388776,"kb_used_data":157400,"kb_used_omap":4,"kb_used_meta":231355,"kb_avail":93983064,"statfs":{"total":96636764160,"available":96238657536,"internally_reserved":0,"allocated":161177600,"data_stored":161192735,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":236908239},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397527,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314448,"kb_used_data":274868,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057392,"statfs":{"total":96636764160,"available":96314769408,"internally_reserved":0,"allocated":281464832,"data_stored":281180447,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234003,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247902,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450968,"kb_used_data":274884,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920872,"statfs":{"total":96636764160,"available":96174972928,"internally_reserved":0,"allocated":281481216,"data_stored":281504031,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":161767424,"data_stored":161771520,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:33:32.649 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 285922, 'reported_epoch': 58, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:33:32.166354+0000', 'last_change': '2026-02-22T23:32:41.678206+0000', 'last_active': '2026-02-22T23:33:32.166354+0000', 'last_peered': '2026-02-22T23:33:32.166354+0000', 'last_clean': '2026-02-22T23:32:35.643295+0000', 'last_became_active': '2026-02-22T23:32:41.677538+0000', 'last_became_peered': '2026-02-22T23:32:41.677538+0000', 'last_unstale': '2026-02-22T23:33:32.166354+0000', 'last_undegraded': '2026-02-22T23:33:32.166354+0000', 'last_fullsized': '2026-02-22T23:33:32.166354+0000', 'mapping_epoch': 57, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 55, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 29110, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 108568, 'num_bytes_recovered': 889381636, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:33:32.649 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:33:37.650 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:33:37.651 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:33:37.652 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:33:37.652 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:33:37.740 INFO:teuthology.orchestra.run.trial115.stdout:98784247904 2026-02-22T23:33:37.740 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:37.742 INFO:teuthology.orchestra.run.trial115.stdout:167503724582 2026-02-22T23:33:37.742 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:37.744 INFO:teuthology.orchestra.run.trial115.stdout:210453397529 2026-02-22T23:33:37.745 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:37.747 INFO:teuthology.orchestra.run.trial115.stdout:231928234005 2026-02-22T23:33:37.748 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:37.893 INFO:teuthology.orchestra.run.trial115.stdout:167503724580 2026-02-22T23:33:37.907 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724582 got 167503724580 for osd.0 2026-02-22T23:33:37.910 INFO:teuthology.orchestra.run.trial115.stdout:98784247903 2026-02-22T23:33:37.922 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247904 got 98784247903 for osd.1 2026-02-22T23:33:37.944 INFO:teuthology.orchestra.run.trial115.stdout:210453397528 2026-02-22T23:33:37.949 INFO:teuthology.orchestra.run.trial115.stdout:231928234004 2026-02-22T23:33:37.957 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397529 got 210453397528 for osd.3 2026-02-22T23:33:37.962 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928234005 got 231928234004 for osd.2 2026-02-22T23:33:38.907 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:38.923 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:38.957 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:38.963 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:39.057 INFO:teuthology.orchestra.run.trial115.stdout:167503724582 2026-02-22T23:33:39.070 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724582 got 167503724582 for osd.0 2026-02-22T23:33:39.070 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:39.109 INFO:teuthology.orchestra.run.trial115.stdout:98784247904 2026-02-22T23:33:39.115 INFO:teuthology.orchestra.run.trial115.stdout:210453397529 2026-02-22T23:33:39.121 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247904 got 98784247904 for osd.1 2026-02-22T23:33:39.121 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:39.128 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397529 got 210453397529 for osd.3 2026-02-22T23:33:39.128 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:39.166 INFO:teuthology.orchestra.run.trial115.stdout:231928234005 2026-02-22T23:33:39.181 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928234005 got 231928234005 for osd.2 2026-02-22T23:33:39.181 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:39.181 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:33:39.363 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:33:39.363 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:33:39.375 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2026-02-22T23:33:38.180342+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":24756,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112924,"num_bytes_recovered":925377284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1187764,"kb_used_data":722676,"kb_used_omap":14,"kb_used_meta":465009,"kb_avail":281927756,"statfs":{"total":289910292480,"available":288694022144,"internally_reserved":0,"allocated":740020224,"data_stored":739773789,"data_compressed":13827,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14890,"internal_metadata":476169686},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-9627,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9627,"num_bytes_recovered":78864384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001329"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":294630,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:33:37.733342+0000","last_change":"2026-02-22T23:32:41.678206+0000","last_active":"2026-02-22T23:33:37.733342+0000","last_peered":"2026-02-22T23:33:37.733342+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:33:37.733342+0000","last_undegraded":"2026-02-22T23:33:37.733342+0000","last_fullsized":"2026-02-22T23:33:37.733342+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":24756,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112922,"num_bytes_recovered":925049604,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":48,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662608+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662608+0000","last_peered":"2026-02-22T23:32:41.662608+0000","last_clean":"2026-02-22T23:32:41.662608+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662608+0000","last_undegraded":"2026-02-22T23:32:41.662608+0000","last_fullsized":"2026-02-22T23:32:41.662608+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":24756,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112922,"num_bytes_recovered":925049604,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1022509056,"data_stored":1022537728,"data_compressed":19575,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724582,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422348,"kb_used_data":172924,"kb_used_omap":4,"kb_used_meta":249403,"kb_avail":93949492,"statfs":{"total":96636764160,"available":96204279808,"internally_reserved":0,"allocated":177074176,"data_stored":177089311,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":255389391},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397529,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314448,"kb_used_data":274868,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057392,"statfs":{"total":96636764160,"available":96314769408,"internally_reserved":0,"allocated":281464832,"data_stored":281180447,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234005,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247904,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450968,"kb_used_data":274884,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920872,"statfs":{"total":96636764160,"available":96174972928,"internally_reserved":0,"allocated":281481216,"data_stored":281504031,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":179568640,"data_stored":179572736,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:33:39.375 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 294630, 'reported_epoch': 58, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-22T23:33:37.733342+0000', 'last_change': '2026-02-22T23:32:41.678206+0000', 'last_active': '2026-02-22T23:33:37.733342+0000', 'last_peered': '2026-02-22T23:33:37.733342+0000', 'last_clean': '2026-02-22T23:32:35.643295+0000', 'last_became_active': '2026-02-22T23:32:41.677538+0000', 'last_became_peered': '2026-02-22T23:32:41.677538+0000', 'last_unstale': '2026-02-22T23:33:37.733342+0000', 'last_undegraded': '2026-02-22T23:33:37.733342+0000', 'last_fullsized': '2026-02-22T23:33:37.733342+0000', 'mapping_epoch': 57, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 55, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 24756, 'num_objects_unfound': 0, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 112922, 'num_bytes_recovered': 925049604, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 68599}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:33:39.375 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-22T23:33:40.906 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:33:40.905+0000 7fa97db55640 -1 osd.3 pg_epoch: 58 pg[2.0s0( v 25'68599 (25'68571,25'68599] local-lis/les=57/58 n=68599 ec=16/16 lis/c=57/54 les/c/f=58/55/0 sis=57) [3,1,0]/[3,1,2]p3(0) backfill=[0(2)] r=0 lpr=57 pi=[54,57)/1 crt=25'68599 lcod 0'0 mlcod 0'0 active+remapped+backfilling rops=1 mbc={0={},1={},2={}}] continue_recovery_op: 2:ad551702:::test:head has inconsistent hinfo 2026-02-22T23:33:40.906 INFO:tasks.ceph.osd.3.trial115.stderr:2026-02-22T23:33:40.905+0000 7fa97db55640 -1 log_channel(cluster) log [ERR] : 2.0s0 missing primary copy of 2:ad551702:::test:head, will try copies on 1(1),2(2) 2026-02-22T23:33:40.907 INFO:tasks.ceph.osd.2.trial115.stderr:2026-02-22T23:33:40.905+0000 7fc092c20640 -1 log_channel(cluster) log [ERR] : Corruption detected: object 2:ad551702:::test:head is missing hash_info 2026-02-22T23:33:44.377 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:33:44.378 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:33:44.378 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:33:44.378 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:33:44.464 INFO:teuthology.orchestra.run.trial115.stdout:98784247907 2026-02-22T23:33:44.465 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:44.466 INFO:teuthology.orchestra.run.trial115.stdout:167503724584 2026-02-22T23:33:44.467 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:44.468 INFO:teuthology.orchestra.run.trial115.stdout:231928234008 2026-02-22T23:33:44.468 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:44.469 INFO:teuthology.orchestra.run.trial115.stdout:210453397531 2026-02-22T23:33:44.470 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:44.660 INFO:teuthology.orchestra.run.trial115.stdout:231928234007 2026-02-22T23:33:44.663 INFO:teuthology.orchestra.run.trial115.stdout:167503724583 2026-02-22T23:33:44.669 INFO:teuthology.orchestra.run.trial115.stdout:210453397530 2026-02-22T23:33:44.669 INFO:teuthology.orchestra.run.trial115.stdout:98784247906 2026-02-22T23:33:44.672 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928234008 got 231928234007 for osd.2 2026-02-22T23:33:44.678 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724584 got 167503724583 for osd.0 2026-02-22T23:33:44.682 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397531 got 210453397530 for osd.3 2026-02-22T23:33:44.684 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247907 got 98784247906 for osd.1 2026-02-22T23:33:45.673 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:45.679 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:45.683 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:45.685 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:45.818 INFO:teuthology.orchestra.run.trial115.stdout:231928234007 2026-02-22T23:33:45.831 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928234008 got 231928234007 for osd.2 2026-02-22T23:33:45.863 INFO:teuthology.orchestra.run.trial115.stdout:98784247906 2026-02-22T23:33:45.872 INFO:teuthology.orchestra.run.trial115.stdout:167503724583 2026-02-22T23:33:45.875 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247907 got 98784247906 for osd.1 2026-02-22T23:33:45.875 INFO:teuthology.orchestra.run.trial115.stdout:210453397530 2026-02-22T23:33:45.884 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724584 got 167503724583 for osd.0 2026-02-22T23:33:45.886 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397531 got 210453397530 for osd.3 2026-02-22T23:33:46.258 INFO:tasks.ceph.mon.a.trial115.stderr:2026-02-22T23:33:46.257+0000 7f2aed71d640 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 1 pg backfill_unfound (PG_DAMAGED) 2026-02-22T23:33:46.832 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:46.875 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:46.885 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:46.887 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:46.991 INFO:teuthology.orchestra.run.trial115.stdout:231928234008 2026-02-22T23:33:47.004 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928234008 got 231928234008 for osd.2 2026-02-22T23:33:47.005 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:47.021 INFO:teuthology.orchestra.run.trial115.stdout:98784247907 2026-02-22T23:33:47.033 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247907 got 98784247907 for osd.1 2026-02-22T23:33:47.033 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:47.066 INFO:teuthology.orchestra.run.trial115.stdout:167503724584 2026-02-22T23:33:47.071 INFO:teuthology.orchestra.run.trial115.stdout:210453397532 2026-02-22T23:33:47.078 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724584 got 167503724584 for osd.0 2026-02-22T23:33:47.078 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:47.083 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397531 got 210453397532 for osd.3 2026-02-22T23:33:47.083 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:47.084 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:33:47.268 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:33:47.268 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:33:47.279 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2026-02-22T23:33:46.181222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":22179,"num_objects_unfound":1,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115498,"num_bytes_recovered":946463492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1214896,"kb_used_data":735920,"kb_used_omap":14,"kb_used_meta":478897,"kb_avail":281900624,"statfs":{"total":289910292480,"available":288666238976,"internally_reserved":0,"allocated":753582080,"data_stored":753335645,"data_compressed":13827,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14890,"internal_metadata":490390998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":-6931,"num_objects_unfound":1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6928,"num_bytes_recovered":56754176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001313"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":299779,"reported_epoch":58,"state":"active+backfill_unfound+remapped","last_fresh":"2026-02-22T23:33:40.905814+0000","last_change":"2026-02-22T23:33:40.906807+0000","last_active":"2026-02-22T23:33:40.905814+0000","last_peered":"2026-02-22T23:33:40.905814+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:33:40.905814+0000","last_undegraded":"2026-02-22T23:33:40.905814+0000","last_fullsized":"2026-02-22T23:33:40.905814+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":22179,"num_objects_unfound":1,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115496,"num_bytes_recovered":946135812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["1(1)"],"object_location_counts":[{"shards":"1(1)","objects":1},{"shards":"1(1),2(2),3(0)","objects":68598}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":22179,"num_objects_unfound":1,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115496,"num_bytes_recovered":946135812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1033068544,"data_stored":1033097216,"data_compressed":19575,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724584,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":449480,"kb_used_data":186168,"kb_used_omap":4,"kb_used_meta":263291,"kb_avail":93922360,"statfs":{"total":96636764160,"available":96176496640,"internally_reserved":0,"allocated":190636032,"data_stored":190651167,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":269610703},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397532,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314448,"kb_used_data":274868,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057392,"statfs":{"total":96636764160,"available":96314769408,"internally_reserved":0,"allocated":281464832,"data_stored":281180447,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234008,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247907,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450968,"kb_used_data":274884,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920872,"statfs":{"total":96636764160,"available":96174972928,"internally_reserved":0,"allocated":281481216,"data_stored":281504031,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":190128128,"data_stored":190132224,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:33:47.280 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 299779, 'reported_epoch': 58, 'state': 'active+backfill_unfound+remapped', 'last_fresh': '2026-02-22T23:33:40.905814+0000', 'last_change': '2026-02-22T23:33:40.906807+0000', 'last_active': '2026-02-22T23:33:40.905814+0000', 'last_peered': '2026-02-22T23:33:40.905814+0000', 'last_clean': '2026-02-22T23:32:35.643295+0000', 'last_became_active': '2026-02-22T23:32:41.677538+0000', 'last_became_peered': '2026-02-22T23:32:41.677538+0000', 'last_unstale': '2026-02-22T23:33:40.905814+0000', 'last_undegraded': '2026-02-22T23:33:40.905814+0000', 'last_fullsized': '2026-02-22T23:33:40.905814+0000', 'mapping_epoch': 57, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 55, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 22179, 'num_objects_unfound': 1, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 115496, 'num_bytes_recovered': 946135812, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['1(1)'], 'object_location_counts': [{'shards': '1(1)', 'objects': 1}, {'shards': '1(1),2(2),3(0)', 'objects': 68598}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:33:47.280 DEBUG:tasks.ec_inconsistent_hinfo:backfilled! 2026-02-22T23:33:47.280 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:33:47.281 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:33:47.282 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:33:47.282 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:33:47.369 INFO:teuthology.orchestra.run.trial115.stdout:210453397533 2026-02-22T23:33:47.370 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:47.372 INFO:teuthology.orchestra.run.trial115.stdout:231928234009 2026-02-22T23:33:47.372 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:47.374 INFO:teuthology.orchestra.run.trial115.stdout:167503724586 2026-02-22T23:33:47.375 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:47.377 INFO:teuthology.orchestra.run.trial115.stdout:98784247908 2026-02-22T23:33:47.377 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:47.525 INFO:teuthology.orchestra.run.trial115.stdout:210453397532 2026-02-22T23:33:47.538 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397533 got 210453397532 for osd.3 2026-02-22T23:33:47.563 INFO:teuthology.orchestra.run.trial115.stdout:231928234008 2026-02-22T23:33:47.566 INFO:teuthology.orchestra.run.trial115.stdout:167503724584 2026-02-22T23:33:47.568 INFO:teuthology.orchestra.run.trial115.stdout:98784247907 2026-02-22T23:33:47.575 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928234009 got 231928234008 for osd.2 2026-02-22T23:33:47.578 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724586 got 167503724584 for osd.0 2026-02-22T23:33:47.581 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247908 got 98784247907 for osd.1 2026-02-22T23:33:48.538 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:48.576 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:48.578 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:48.581 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:48.698 INFO:teuthology.orchestra.run.trial115.stdout:210453397533 2026-02-22T23:33:48.710 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397533 got 210453397533 for osd.3 2026-02-22T23:33:48.711 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:48.725 INFO:teuthology.orchestra.run.trial115.stdout:231928234009 2026-02-22T23:33:48.736 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928234009 got 231928234009 for osd.2 2026-02-22T23:33:48.736 INFO:teuthology.orchestra.run.trial115.stdout:98784247908 2026-02-22T23:33:48.736 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:48.739 INFO:teuthology.orchestra.run.trial115.stdout:167503724586 2026-02-22T23:33:48.749 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247908 got 98784247908 for osd.1 2026-02-22T23:33:48.749 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:48.751 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724586 got 167503724586 for osd.0 2026-02-22T23:33:48.751 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:48.751 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:33:48.937 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:33:48.937 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:33:48.947 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2026-02-22T23:33:48.181495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":22179,"num_objects_unfound":1,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115498,"num_bytes_recovered":946463492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1214896,"kb_used_data":735920,"kb_used_omap":14,"kb_used_meta":478897,"kb_avail":281900624,"statfs":{"total":289910292480,"available":288666238976,"internally_reserved":0,"allocated":753582080,"data_stored":753335645,"data_compressed":13827,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14890,"internal_metadata":490390998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":-4265,"num_objects_unfound":1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4262,"num_bytes_recovered":34914304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001360"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":299779,"reported_epoch":58,"state":"active+backfill_unfound+remapped","last_fresh":"2026-02-22T23:33:40.905814+0000","last_change":"2026-02-22T23:33:40.906807+0000","last_active":"2026-02-22T23:33:40.905814+0000","last_peered":"2026-02-22T23:33:40.905814+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:33:40.905814+0000","last_undegraded":"2026-02-22T23:33:40.905814+0000","last_fullsized":"2026-02-22T23:33:40.905814+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":22179,"num_objects_unfound":1,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115496,"num_bytes_recovered":946135812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["1(1)"],"object_location_counts":[{"shards":"1(1)","objects":1},{"shards":"1(1),2(2),3(0)","objects":68598}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":22179,"num_objects_unfound":1,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115496,"num_bytes_recovered":946135812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1033068544,"data_stored":1033097216,"data_compressed":19575,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724586,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":449480,"kb_used_data":186168,"kb_used_omap":4,"kb_used_meta":263291,"kb_avail":93922360,"statfs":{"total":96636764160,"available":96176496640,"internally_reserved":0,"allocated":190636032,"data_stored":190651167,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":269610703},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397533,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314448,"kb_used_data":274868,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057392,"statfs":{"total":96636764160,"available":96314769408,"internally_reserved":0,"allocated":281464832,"data_stored":281180447,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234009,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247908,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450968,"kb_used_data":274884,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920872,"statfs":{"total":96636764160,"available":96174972928,"internally_reserved":0,"allocated":281481216,"data_stored":281504031,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":190128128,"data_stored":190132224,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:33:48.948 DEBUG:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "25'68599", 'reported_seq': 299779, 'reported_epoch': 58, 'state': 'active+backfill_unfound+remapped', 'last_fresh': '2026-02-22T23:33:40.905814+0000', 'last_change': '2026-02-22T23:33:40.906807+0000', 'last_active': '2026-02-22T23:33:40.905814+0000', 'last_peered': '2026-02-22T23:33:40.905814+0000', 'last_clean': '2026-02-22T23:32:35.643295+0000', 'last_became_active': '2026-02-22T23:32:41.677538+0000', 'last_became_peered': '2026-02-22T23:32:41.677538+0000', 'last_unstale': '2026-02-22T23:33:40.905814+0000', 'last_undegraded': '2026-02-22T23:33:40.905814+0000', 'last_fullsized': '2026-02-22T23:33:40.905814+0000', 'mapping_epoch': 57, 'log_start': "25'68571", 'ondisk_log_start': "25'68571", 'created': 16, 'last_epoch_clean': 55, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'last_clean_scrub_stamp': '2026-02-22T23:28:22.258373+0000', 'objects_scrubbed': 2, 'log_size': 28, 'log_dups_size': 3000, 'ondisk_log_size': 28, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 561955214, 'num_objects': 68599, 'num_object_clones': 0, 'num_object_copies': 205797, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 22179, 'num_objects_unfound': 1, 'num_objects_dirty': 68599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 68599, 'num_write_kb': 548787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 115496, 'num_bytes_recovered': 946135812, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['1(1)'], 'object_location_counts': [{'shards': '1(1)', 'objects': 1}, {'shards': '1(1),2(2),3(0)', 'objects': 68598}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-22T23:33:48.948 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-22T23:33:49.129 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:33:49.129 INFO:teuthology.orchestra.run.trial115.stdout:{"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"1/68601 objects unfound (0.001%)","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 1 pg backfill_unfound","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":385,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":58,"num_osds":4,"num_up_osds":4,"osd_up_since":1771803154,"num_in_osds":3,"osd_in_since":1771803159,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+backfill_unfound+remapped","count":1}],"num_pgs":2,"num_pools":2,"num_objects":68601,"data_bytes":562283406,"bytes_used":1244053504,"bytes_avail":288666238976,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":205801,"degraded_ratio":9.7181257622654892e-06,"misplaced_objects":22179,"misplaced_total":205801,"misplaced_ratio":0.10776915564064314,"unfound_objects":1,"unfound_total":68601,"unfound_ratio":1.4577046981822422e-05,"recovering_objects_per_sec":355,"recovering_bytes_per_sec":2909195,"recovering_keys_per_sec":0,"num_objects_recovered":4262,"num_bytes_recovered":34914304,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2026-02-22T23:27:24:013195+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-22T23:33:34.179945+0000","services":{}},"progress_events":{}} 2026-02-22T23:33:49.140 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:{'fsid': 'a4a3ada6-bd8c-4e08-a984-e18b05a50ce6', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '1/68601 objects unfound (0.001%)', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 1 pg backfill_unfound', 'count': 1}, 'muted': False}}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 385, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 58, 'num_osds': 4, 'num_up_osds': 4, 'osd_up_since': 1771803154, 'num_in_osds': 3, 'osd_in_since': 1771803159, 'num_remapped_pgs': 1}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 1}, {'state_name': 'active+backfill_unfound+remapped', 'count': 1}], 'num_pgs': 2, 'num_pools': 2, 'num_objects': 68601, 'data_bytes': 562283406, 'bytes_used': 1244053504, 'bytes_avail': 288666238976, 'bytes_total': 289910292480, 'degraded_objects': 2, 'degraded_total': 205801, 'degraded_ratio': 9.71812576226549e-06, 'misplaced_objects': 22179, 'misplaced_total': 205801, 'misplaced_ratio': 0.10776915564064314, 'unfound_objects': 1, 'unfound_total': 68601, 'unfound_ratio': 1.4577046981822422e-05, 'recovering_objects_per_sec': 355, 'recovering_bytes_per_sec': 2909195, 'recovering_keys_per_sec': 0, 'num_objects_recovered': 4262, 'num_bytes_recovered': 34914304, 'num_keys_recovered': 0}, 'fsmap': {'epoch': 1, 'btime': '2026-02-22T23:27:24:013195+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 8, 'modified': '2026-02-22T23:33:34.179945+0000', 'services': {}}, 'progress_events': {}} 2026-02-22T23:33:49.140 DEBUG:tasks.ec_inconsistent_hinfo:there are 1 unfound objects 2026-02-22T23:33:49.140 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.0 list_unfound '{}' 2026-02-22T23:33:49.213 INFO:teuthology.orchestra.run.trial115.stdout:{ 2026-02-22T23:33:49.213 INFO:teuthology.orchestra.run.trial115.stdout: "offset": { 2026-02-22T23:33:49.213 INFO:teuthology.orchestra.run.trial115.stdout: "oid": "", 2026-02-22T23:33:49.214 INFO:teuthology.orchestra.run.trial115.stdout: "key": "", 2026-02-22T23:33:49.214 INFO:teuthology.orchestra.run.trial115.stdout: "snapid": 0, 2026-02-22T23:33:49.214 INFO:teuthology.orchestra.run.trial115.stdout: "hash": 0, 2026-02-22T23:33:49.214 INFO:teuthology.orchestra.run.trial115.stdout: "max": 0, 2026-02-22T23:33:49.214 INFO:teuthology.orchestra.run.trial115.stdout: "pool": -9223372036854775808, 2026-02-22T23:33:49.214 INFO:teuthology.orchestra.run.trial115.stdout: "namespace": "" 2026-02-22T23:33:49.214 INFO:teuthology.orchestra.run.trial115.stdout: }, 2026-02-22T23:33:49.214 INFO:teuthology.orchestra.run.trial115.stdout: "num_missing": 1, 2026-02-22T23:33:49.214 INFO:teuthology.orchestra.run.trial115.stdout: "num_unfound": 1, 2026-02-22T23:33:49.214 INFO:teuthology.orchestra.run.trial115.stdout: "objects": [ 2026-02-22T23:33:49.214 INFO:teuthology.orchestra.run.trial115.stdout: { 2026-02-22T23:33:49.214 INFO:teuthology.orchestra.run.trial115.stdout: "oid": { 2026-02-22T23:33:49.215 INFO:teuthology.orchestra.run.trial115.stdout: "oid": "test", 2026-02-22T23:33:49.215 INFO:teuthology.orchestra.run.trial115.stdout: "key": "", 2026-02-22T23:33:49.215 INFO:teuthology.orchestra.run.trial115.stdout: "snapid": -2, 2026-02-22T23:33:49.215 INFO:teuthology.orchestra.run.trial115.stdout: "hash": 1088989877, 2026-02-22T23:33:49.215 INFO:teuthology.orchestra.run.trial115.stdout: "max": 0, 2026-02-22T23:33:49.215 INFO:teuthology.orchestra.run.trial115.stdout: "pool": 2, 2026-02-22T23:33:49.215 INFO:teuthology.orchestra.run.trial115.stdout: "namespace": "" 2026-02-22T23:33:49.215 INFO:teuthology.orchestra.run.trial115.stdout: }, 2026-02-22T23:33:49.215 INFO:teuthology.orchestra.run.trial115.stdout: "need": "19'2", 2026-02-22T23:33:49.215 INFO:teuthology.orchestra.run.trial115.stdout: "have": "0'0", 2026-02-22T23:33:49.215 INFO:teuthology.orchestra.run.trial115.stdout: "flags": "none", 2026-02-22T23:33:49.215 INFO:teuthology.orchestra.run.trial115.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2026-02-22T23:33:49.216 INFO:teuthology.orchestra.run.trial115.stdout: "locations": [ 2026-02-22T23:33:49.216 INFO:teuthology.orchestra.run.trial115.stdout: "1(1)" 2026-02-22T23:33:49.216 INFO:teuthology.orchestra.run.trial115.stdout: ] 2026-02-22T23:33:49.216 INFO:teuthology.orchestra.run.trial115.stdout: } 2026-02-22T23:33:49.216 INFO:teuthology.orchestra.run.trial115.stdout: ], 2026-02-22T23:33:49.216 INFO:teuthology.orchestra.run.trial115.stdout: "state": "NotBackfilling", 2026-02-22T23:33:49.216 INFO:teuthology.orchestra.run.trial115.stdout: "available_might_have_unfound": true, 2026-02-22T23:33:49.216 INFO:teuthology.orchestra.run.trial115.stdout: "might_have_unfound": [], 2026-02-22T23:33:49.216 INFO:teuthology.orchestra.run.trial115.stdout: "more": false 2026-02-22T23:33:49.216 INFO:teuthology.orchestra.run.trial115.stdout:} 2026-02-22T23:33:49.223 DEBUG:tasks.ec_inconsistent_hinfo:list_pg_unfound={'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'test', 'key': '', 'snapid': -2, 'hash': 1088989877, 'max': 0, 'pool': 2, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['1(1)']}], 'state': 'NotBackfilling', 'available_might_have_unfound': True, 'might_have_unfound': []} 2026-02-22T23:33:49.223 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:33:49.376 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:33:49.376 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:33:49.387 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2026-02-22T23:33:48.181495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":22179,"num_objects_unfound":1,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115498,"num_bytes_recovered":946463492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1214896,"kb_used_data":735920,"kb_used_omap":14,"kb_used_meta":478897,"kb_avail":281900624,"statfs":{"total":289910292480,"available":288666238976,"internally_reserved":0,"allocated":753582080,"data_stored":753335645,"data_compressed":13827,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14890,"internal_metadata":490390998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":-4265,"num_objects_unfound":1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4262,"num_bytes_recovered":34914304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001360"},"pg_stats":[{"pgid":"2.0","version":"25'68599","reported_seq":299779,"reported_epoch":58,"state":"active+backfill_unfound+remapped","last_fresh":"2026-02-22T23:33:40.905814+0000","last_change":"2026-02-22T23:33:40.906807+0000","last_active":"2026-02-22T23:33:40.905814+0000","last_peered":"2026-02-22T23:33:40.905814+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:33:40.905814+0000","last_undegraded":"2026-02-22T23:33:40.905814+0000","last_fullsized":"2026-02-22T23:33:40.905814+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":3000,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":22179,"num_objects_unfound":1,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115496,"num_bytes_recovered":946135812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["1(1)"],"object_location_counts":[{"shards":"1(1)","objects":1},{"shards":"1(1),2(2),3(0)","objects":68598}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":22179,"num_objects_unfound":1,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115496,"num_bytes_recovered":946135812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1033068544,"data_stored":1033097216,"data_compressed":19575,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1695,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724586,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":449480,"kb_used_data":186168,"kb_used_omap":4,"kb_used_meta":263291,"kb_avail":93922360,"statfs":{"total":96636764160,"available":96176496640,"internally_reserved":0,"allocated":190636032,"data_stored":190651167,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":269610703},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397533,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314448,"kb_used_data":274868,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057392,"statfs":{"total":96636764160,"available":96314769408,"internally_reserved":0,"allocated":281464832,"data_stored":281180447,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234009,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247908,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450968,"kb_used_data":274884,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920872,"statfs":{"total":96636764160,"available":96174972928,"internally_reserved":0,"allocated":281481216,"data_stored":281504031,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":190128128,"data_stored":190132224,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280985600,"data_compressed":4262,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":6934,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280981504,"data_stored":280989696,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:33:49.388 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.0 mark_unfound_lost delete 2026-02-22T23:33:49.469 INFO:teuthology.orchestra.run.trial115.stderr:pg has 1 objects unfound and apparently lost marking 2026-02-22T23:33:49.480 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-22T23:33:49.480 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-22T23:33:49.481 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-22T23:33:49.481 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-22T23:33:49.565 INFO:teuthology.orchestra.run.trial115.stdout:231928234011 2026-02-22T23:33:49.565 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:49.566 INFO:teuthology.orchestra.run.trial115.stdout:167503724587 2026-02-22T23:33:49.567 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:49.568 INFO:teuthology.orchestra.run.trial115.stdout:98784247910 2026-02-22T23:33:49.568 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:49.576 INFO:teuthology.orchestra.run.trial115.stdout:210453397534 2026-02-22T23:33:49.576 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:49.719 INFO:teuthology.orchestra.run.trial115.stdout:231928234009 2026-02-22T23:33:49.732 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928234011 got 231928234009 for osd.2 2026-02-22T23:33:49.769 INFO:teuthology.orchestra.run.trial115.stdout:167503724586 2026-02-22T23:33:49.773 INFO:teuthology.orchestra.run.trial115.stdout:98784247908 2026-02-22T23:33:49.778 INFO:teuthology.orchestra.run.trial115.stdout:210453397533 2026-02-22T23:33:49.781 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724587 got 167503724586 for osd.0 2026-02-22T23:33:49.785 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247910 got 98784247908 for osd.1 2026-02-22T23:33:49.791 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397534 got 210453397533 for osd.3 2026-02-22T23:33:50.733 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-22T23:33:50.782 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-22T23:33:50.785 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-22T23:33:50.792 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-22T23:33:50.895 INFO:teuthology.orchestra.run.trial115.stdout:231928234011 2026-02-22T23:33:50.908 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 231928234011 got 231928234011 for osd.2 2026-02-22T23:33:50.908 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:50.926 INFO:teuthology.orchestra.run.trial115.stdout:167503724587 2026-02-22T23:33:50.938 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 167503724587 got 167503724587 for osd.0 2026-02-22T23:33:50.938 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:50.982 INFO:teuthology.orchestra.run.trial115.stdout:210453397534 2026-02-22T23:33:50.985 INFO:teuthology.orchestra.run.trial115.stdout:98784247910 2026-02-22T23:33:50.994 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 210453397534 got 210453397534 for osd.3 2026-02-22T23:33:50.994 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:50.999 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247910 got 98784247910 for osd.1 2026-02-22T23:33:50.999 DEBUG:teuthology.parallel:result is None 2026-02-22T23:33:50.999 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for recovery to complete 2026-02-22T23:33:50.999 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:33:51.186 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:33:51.186 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:33:51.198 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2026-02-22T23:33:50.181810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22101,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115579,"num_bytes_recovered":947118852,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1214896,"kb_used_data":735920,"kb_used_omap":14,"kb_used_meta":478897,"kb_avail":281900624,"statfs":{"total":289910292480,"available":288666238976,"internally_reserved":0,"allocated":753582080,"data_stored":753335645,"data_compressed":13827,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14890,"internal_metadata":490390998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2655,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2655,"num_bytes_recovered":21741568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001468"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":299948,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:33:49.562944+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:33:49.562944+0000","last_peered":"2026-02-22T23:33:49.562944+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:33:49.562944+0000","last_undegraded":"2026-02-22T23:33:49.562944+0000","last_fullsized":"2026-02-22T23:33:49.562944+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22101,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115577,"num_bytes_recovered":946791172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22101,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115577,"num_bytes_recovered":946791172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1033338880,"data_stored":1033355264,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724587,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":449480,"kb_used_data":186168,"kb_used_omap":4,"kb_used_meta":263291,"kb_avail":93922360,"statfs":{"total":96636764160,"available":96176496640,"internally_reserved":0,"allocated":190636032,"data_stored":190651167,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":269610703},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397534,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314448,"kb_used_data":274868,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057392,"statfs":{"total":96636764160,"available":96314769408,"internally_reserved":0,"allocated":281464832,"data_stored":281180447,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234011,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247910,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450968,"kb_used_data":274884,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920872,"statfs":{"total":96636764160,"available":96174972928,"internally_reserved":0,"allocated":281481216,"data_stored":281504031,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":190410752,"data_stored":190414848,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:33:51.199 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:33:51.340 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:33:51.340 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:33:51.352 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2026-02-22T23:33:50.181810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22101,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115579,"num_bytes_recovered":947118852,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1214896,"kb_used_data":735920,"kb_used_omap":14,"kb_used_meta":478897,"kb_avail":281900624,"statfs":{"total":289910292480,"available":288666238976,"internally_reserved":0,"allocated":753582080,"data_stored":753335645,"data_compressed":13827,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14890,"internal_metadata":490390998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2655,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2655,"num_bytes_recovered":21741568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001468"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":299948,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:33:49.562944+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:33:49.562944+0000","last_peered":"2026-02-22T23:33:49.562944+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:33:49.562944+0000","last_undegraded":"2026-02-22T23:33:49.562944+0000","last_fullsized":"2026-02-22T23:33:49.562944+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22101,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115577,"num_bytes_recovered":946791172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22101,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115577,"num_bytes_recovered":946791172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1033338880,"data_stored":1033355264,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724587,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":449480,"kb_used_data":186168,"kb_used_omap":4,"kb_used_meta":263291,"kb_avail":93922360,"statfs":{"total":96636764160,"available":96176496640,"internally_reserved":0,"allocated":190636032,"data_stored":190651167,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":269610703},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397534,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314448,"kb_used_data":274868,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057392,"statfs":{"total":96636764160,"available":96314769408,"internally_reserved":0,"allocated":281464832,"data_stored":281180447,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234011,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247910,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450968,"kb_used_data":274884,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920872,"statfs":{"total":96636764160,"available":96174972928,"internally_reserved":0,"allocated":281481216,"data_stored":281504031,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":190410752,"data_stored":190414848,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:33:51.353 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:33:51.495 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:33:51.496 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:33:51.506 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2026-02-22T23:33:50.181810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22101,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115579,"num_bytes_recovered":947118852,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1214896,"kb_used_data":735920,"kb_used_omap":14,"kb_used_meta":478897,"kb_avail":281900624,"statfs":{"total":289910292480,"available":288666238976,"internally_reserved":0,"allocated":753582080,"data_stored":753335645,"data_compressed":13827,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14890,"internal_metadata":490390998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2655,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2655,"num_bytes_recovered":21741568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001468"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":299948,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:33:49.562944+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:33:49.562944+0000","last_peered":"2026-02-22T23:33:49.562944+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:33:49.562944+0000","last_undegraded":"2026-02-22T23:33:49.562944+0000","last_fullsized":"2026-02-22T23:33:49.562944+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22101,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115577,"num_bytes_recovered":946791172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22101,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115577,"num_bytes_recovered":946791172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1033338880,"data_stored":1033355264,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724587,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":449480,"kb_used_data":186168,"kb_used_omap":4,"kb_used_meta":263291,"kb_avail":93922360,"statfs":{"total":96636764160,"available":96176496640,"internally_reserved":0,"allocated":190636032,"data_stored":190651167,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":269610703},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397534,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314448,"kb_used_data":274868,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057392,"statfs":{"total":96636764160,"available":96314769408,"internally_reserved":0,"allocated":281464832,"data_stored":281180447,"data_compressed":4912,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234011,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247910,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450968,"kb_used_data":274884,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920872,"statfs":{"total":96636764160,"available":96174972928,"internally_reserved":0,"allocated":281481216,"data_stored":281504031,"data_compressed":4855,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":190410752,"data_stored":190414848,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:33:54.507 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:33:54.659 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:33:54.659 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:33:54.670 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2026-02-22T23:33:54.182293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":21340,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116340,"num_bytes_recovered":953352964,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1227536,"kb_used_data":740256,"kb_used_omap":14,"kb_used_meta":487217,"kb_avail":281887984,"statfs":{"total":289910292480,"available":288653295616,"internally_reserved":0,"allocated":758022144,"data_stored":757767517,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":498910678},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1141,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1141,"num_bytes_recovered":9338880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001470"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":301470,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:33:50.537621+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:33:50.537621+0000","last_peered":"2026-02-22T23:33:50.537621+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:33:50.537621+0000","last_undegraded":"2026-02-22T23:33:50.537621+0000","last_fullsized":"2026-02-22T23:33:50.537621+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":21340,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116338,"num_bytes_recovered":953025284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":21340,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116338,"num_bytes_recovered":953025284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1038880768,"data_stored":1038897152,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462144,"kb_used_data":190512,"kb_used_omap":4,"kb_used_meta":271611,"kb_avail":93909696,"statfs":{"total":96636764160,"available":96163528704,"internally_reserved":0,"allocated":195084288,"data_stored":195099423,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":278130383},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397535,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":274864,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":281460736,"data_stored":281172255,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450960,"kb_used_data":274880,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920880,"statfs":{"total":96636764160,"available":96174981120,"internally_reserved":0,"allocated":281477120,"data_stored":281495839,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":195952640,"data_stored":195956736,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:33:54.671 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:33:54.817 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:33:54.818 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:33:54.829 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2026-02-22T23:33:54.182293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":21340,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116340,"num_bytes_recovered":953352964,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1227536,"kb_used_data":740256,"kb_used_omap":14,"kb_used_meta":487217,"kb_avail":281887984,"statfs":{"total":289910292480,"available":288653295616,"internally_reserved":0,"allocated":758022144,"data_stored":757767517,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":498910678},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1141,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1141,"num_bytes_recovered":9338880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001470"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":301470,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:33:50.537621+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:33:50.537621+0000","last_peered":"2026-02-22T23:33:50.537621+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:33:50.537621+0000","last_undegraded":"2026-02-22T23:33:50.537621+0000","last_fullsized":"2026-02-22T23:33:50.537621+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":21340,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116338,"num_bytes_recovered":953025284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":21340,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116338,"num_bytes_recovered":953025284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1038880768,"data_stored":1038897152,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462144,"kb_used_data":190512,"kb_used_omap":4,"kb_used_meta":271611,"kb_avail":93909696,"statfs":{"total":96636764160,"available":96163528704,"internally_reserved":0,"allocated":195084288,"data_stored":195099423,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":278130383},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397535,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":274864,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":281460736,"data_stored":281172255,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450960,"kb_used_data":274880,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920880,"statfs":{"total":96636764160,"available":96174981120,"internally_reserved":0,"allocated":281477120,"data_stored":281495839,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":195952640,"data_stored":195956736,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:33:57.829 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:33:57.971 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:33:57.971 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:33:57.981 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2026-02-22T23:33:56.182512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17646,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120034,"num_bytes_recovered":983606044,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1227536,"kb_used_data":740256,"kb_used_omap":14,"kb_used_meta":487217,"kb_avail":281887984,"statfs":{"total":289910292480,"available":288653295616,"internally_reserved":0,"allocated":758022144,"data_stored":757767517,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":498910678},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-4835,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4835,"num_bytes_recovered":39591960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001484"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":308858,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:33:55.538553+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:33:55.538553+0000","last_peered":"2026-02-22T23:33:55.538553+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:33:55.538553+0000","last_undegraded":"2026-02-22T23:33:55.538553+0000","last_fullsized":"2026-02-22T23:33:55.538553+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17646,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120032,"num_bytes_recovered":983278364,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17646,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120032,"num_bytes_recovered":983278364,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1038880768,"data_stored":1038897152,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462144,"kb_used_data":190512,"kb_used_omap":4,"kb_used_meta":271611,"kb_avail":93909696,"statfs":{"total":96636764160,"available":96163528704,"internally_reserved":0,"allocated":195084288,"data_stored":195099423,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":278130383},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397536,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":274864,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":281460736,"data_stored":281172255,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450960,"kb_used_data":274880,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920880,"statfs":{"total":96636764160,"available":96174981120,"internally_reserved":0,"allocated":281477120,"data_stored":281495839,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":195952640,"data_stored":195956736,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:33:57.982 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:33:58.131 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:33:58.132 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:33:58.144 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2026-02-22T23:33:56.182512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17646,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120034,"num_bytes_recovered":983606044,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1227536,"kb_used_data":740256,"kb_used_omap":14,"kb_used_meta":487217,"kb_avail":281887984,"statfs":{"total":289910292480,"available":288653295616,"internally_reserved":0,"allocated":758022144,"data_stored":757767517,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":498910678},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-4835,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4835,"num_bytes_recovered":39591960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001484"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":308858,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:33:55.538553+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:33:55.538553+0000","last_peered":"2026-02-22T23:33:55.538553+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:33:55.538553+0000","last_undegraded":"2026-02-22T23:33:55.538553+0000","last_fullsized":"2026-02-22T23:33:55.538553+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17646,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120032,"num_bytes_recovered":983278364,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17646,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120032,"num_bytes_recovered":983278364,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1038880768,"data_stored":1038897152,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462144,"kb_used_data":190512,"kb_used_omap":4,"kb_used_meta":271611,"kb_avail":93909696,"statfs":{"total":96636764160,"available":96163528704,"internally_reserved":0,"allocated":195084288,"data_stored":195099423,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":278130383},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397536,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":274864,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":281460736,"data_stored":281172255,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450960,"kb_used_data":274880,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920880,"statfs":{"total":96636764160,"available":96174981120,"internally_reserved":0,"allocated":281477120,"data_stored":281495839,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":195952640,"data_stored":195956736,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:01.146 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:01.299 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:01.299 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:01.311 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2026-02-22T23:34:00.182950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17646,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120034,"num_bytes_recovered":983606044,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1274264,"kb_used_data":755048,"kb_used_omap":14,"kb_used_meta":519153,"kb_avail":281841256,"statfs":{"total":289910292480,"available":288605446144,"internally_reserved":0,"allocated":773169152,"data_stored":772914525,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":531613142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":-1,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":-4533,"num_objects_unfound":-1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4536,"num_bytes_recovered":37142552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001455"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":308858,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:33:55.538553+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:33:55.538553+0000","last_peered":"2026-02-22T23:33:55.538553+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:33:55.538553+0000","last_undegraded":"2026-02-22T23:33:55.538553+0000","last_fullsized":"2026-02-22T23:33:55.538553+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17646,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120032,"num_bytes_recovered":983278364,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17646,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120032,"num_bytes_recovered":983278364,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1053913088,"data_stored":1053929472,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724589,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":508872,"kb_used_data":205304,"kb_used_omap":4,"kb_used_meta":303547,"kb_avail":93862968,"statfs":{"total":96636764160,"available":96115679232,"internally_reserved":0,"allocated":210231296,"data_stored":210246431,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":310832847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397536,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":274864,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":281460736,"data_stored":281172255,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234013,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450960,"kb_used_data":274880,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920880,"statfs":{"total":96636764160,"available":96174981120,"internally_reserved":0,"allocated":281477120,"data_stored":281495839,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":210984960,"data_stored":210989056,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:01.312 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:01.453 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:01.453 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:01.463 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2026-02-22T23:34:00.182950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17646,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120034,"num_bytes_recovered":983606044,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1274264,"kb_used_data":755048,"kb_used_omap":14,"kb_used_meta":519153,"kb_avail":281841256,"statfs":{"total":289910292480,"available":288605446144,"internally_reserved":0,"allocated":773169152,"data_stored":772914525,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":531613142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":-1,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":-4533,"num_objects_unfound":-1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4536,"num_bytes_recovered":37142552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001455"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":308858,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:33:55.538553+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:33:55.538553+0000","last_peered":"2026-02-22T23:33:55.538553+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:33:55.538553+0000","last_undegraded":"2026-02-22T23:33:55.538553+0000","last_fullsized":"2026-02-22T23:33:55.538553+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17646,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120032,"num_bytes_recovered":983278364,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17646,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120032,"num_bytes_recovered":983278364,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1053913088,"data_stored":1053929472,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724589,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":508872,"kb_used_data":205304,"kb_used_omap":4,"kb_used_meta":303547,"kb_avail":93862968,"statfs":{"total":96636764160,"available":96115679232,"internally_reserved":0,"allocated":210231296,"data_stored":210246431,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":310832847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397536,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":274864,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":281460736,"data_stored":281172255,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234013,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450960,"kb_used_data":274880,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920880,"statfs":{"total":96636764160,"available":96174981120,"internally_reserved":0,"allocated":281477120,"data_stored":281495839,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":210984960,"data_stored":210989056,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:04.465 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:04.615 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:04.615 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:04.627 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2026-02-22T23:34:04.183391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13779,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123901,"num_bytes_recovered":1015284508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1289868,"kb_used_data":770652,"kb_used_omap":14,"kb_used_meta":519153,"kb_avail":281825652,"statfs":{"total":289910292480,"available":288589467648,"internally_reserved":0,"allocated":789147648,"data_stored":788893021,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":531613142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-7561,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7561,"num_bytes_recovered":61931544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001313"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":316592,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:34:00.539137+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:34:00.539137+0000","last_peered":"2026-02-22T23:34:00.539137+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:34:00.539137+0000","last_undegraded":"2026-02-22T23:34:00.539137+0000","last_fullsized":"2026-02-22T23:34:00.539137+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13779,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123899,"num_bytes_recovered":1014956828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13779,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123899,"num_bytes_recovered":1014956828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1069793280,"data_stored":1069809664,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":524476,"kb_used_data":220908,"kb_used_omap":4,"kb_used_meta":303547,"kb_avail":93847364,"statfs":{"total":96636764160,"available":96099700736,"internally_reserved":0,"allocated":226209792,"data_stored":226224927,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":310832847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397537,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":274864,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":281460736,"data_stored":281172255,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450960,"kb_used_data":274880,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920880,"statfs":{"total":96636764160,"available":96174981120,"internally_reserved":0,"allocated":281477120,"data_stored":281495839,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":226865152,"data_stored":226869248,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:04.627 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:04.770 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:04.770 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:04.781 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2026-02-22T23:34:04.183391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13779,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123901,"num_bytes_recovered":1015284508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1289868,"kb_used_data":770652,"kb_used_omap":14,"kb_used_meta":519153,"kb_avail":281825652,"statfs":{"total":289910292480,"available":288589467648,"internally_reserved":0,"allocated":789147648,"data_stored":788893021,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":531613142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-7561,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7561,"num_bytes_recovered":61931544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001313"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":316592,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:34:00.539137+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:34:00.539137+0000","last_peered":"2026-02-22T23:34:00.539137+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:34:00.539137+0000","last_undegraded":"2026-02-22T23:34:00.539137+0000","last_fullsized":"2026-02-22T23:34:00.539137+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13779,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123899,"num_bytes_recovered":1014956828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13779,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123899,"num_bytes_recovered":1014956828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1069793280,"data_stored":1069809664,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":524476,"kb_used_data":220908,"kb_used_omap":4,"kb_used_meta":303547,"kb_avail":93847364,"statfs":{"total":96636764160,"available":96099700736,"internally_reserved":0,"allocated":226209792,"data_stored":226224927,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":310832847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397537,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":274864,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":281460736,"data_stored":281172255,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450960,"kb_used_data":274880,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920880,"statfs":{"total":96636764160,"available":96174981120,"internally_reserved":0,"allocated":281477120,"data_stored":281495839,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":226865152,"data_stored":226869248,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:07.783 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:07.926 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:07.926 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:07.937 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2026-02-22T23:34:06.183600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9751,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127929,"num_bytes_recovered":1048281884,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1289868,"kb_used_data":770652,"kb_used_omap":14,"kb_used_meta":519153,"kb_avail":281825652,"statfs":{"total":289910292480,"available":288589467648,"internally_reserved":0,"allocated":789147648,"data_stored":788893021,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":531613142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-11589,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11589,"num_bytes_recovered":94928920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001307"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":324648,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:34:05.539007+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:34:05.539007+0000","last_peered":"2026-02-22T23:34:05.539007+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:34:05.539007+0000","last_undegraded":"2026-02-22T23:34:05.539007+0000","last_fullsized":"2026-02-22T23:34:05.539007+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9751,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127927,"num_bytes_recovered":1047954204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9751,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127927,"num_bytes_recovered":1047954204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1069793280,"data_stored":1069809664,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":524476,"kb_used_data":220908,"kb_used_omap":4,"kb_used_meta":303547,"kb_avail":93847364,"statfs":{"total":96636764160,"available":96099700736,"internally_reserved":0,"allocated":226209792,"data_stored":226224927,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":310832847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397538,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":274864,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":281460736,"data_stored":281172255,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450960,"kb_used_data":274880,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920880,"statfs":{"total":96636764160,"available":96174981120,"internally_reserved":0,"allocated":281477120,"data_stored":281495839,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":226865152,"data_stored":226869248,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:07.938 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:08.078 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:08.078 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:08.088 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2026-02-22T23:34:06.183600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9751,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127929,"num_bytes_recovered":1048281884,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1289868,"kb_used_data":770652,"kb_used_omap":14,"kb_used_meta":519153,"kb_avail":281825652,"statfs":{"total":289910292480,"available":288589467648,"internally_reserved":0,"allocated":789147648,"data_stored":788893021,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":531613142},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-11589,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11589,"num_bytes_recovered":94928920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001307"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":324648,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:34:05.539007+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:34:05.539007+0000","last_peered":"2026-02-22T23:34:05.539007+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:34:05.539007+0000","last_undegraded":"2026-02-22T23:34:05.539007+0000","last_fullsized":"2026-02-22T23:34:05.539007+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9751,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127927,"num_bytes_recovered":1047954204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9751,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127927,"num_bytes_recovered":1047954204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1069793280,"data_stored":1069809664,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":524476,"kb_used_data":220908,"kb_used_omap":4,"kb_used_meta":303547,"kb_avail":93847364,"statfs":{"total":96636764160,"available":96099700736,"internally_reserved":0,"allocated":226209792,"data_stored":226224927,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":310832847},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397538,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":274864,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":281460736,"data_stored":281172255,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450960,"kb_used_data":274880,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920880,"statfs":{"total":96636764160,"available":96174981120,"internally_reserved":0,"allocated":281477120,"data_stored":281495839,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":226865152,"data_stored":226869248,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:11.090 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:11.240 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:11.241 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:11.250 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2026-02-22T23:34:10.184121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9751,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127929,"num_bytes_recovered":1048281884,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1342096,"kb_used_data":786784,"kb_used_omap":14,"kb_used_meta":555249,"kb_avail":281773424,"statfs":{"total":289910292480,"available":288535986176,"internally_reserved":0,"allocated":805666816,"data_stored":805412189,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":568575446},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-7895,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7895,"num_bytes_recovered":64675840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001389"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":324648,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:34:05.539007+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:34:05.539007+0000","last_peered":"2026-02-22T23:34:05.539007+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:34:05.539007+0000","last_undegraded":"2026-02-22T23:34:05.539007+0000","last_fullsized":"2026-02-22T23:34:05.539007+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9751,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127927,"num_bytes_recovered":1047954204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9751,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127927,"num_bytes_recovered":1047954204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1086414848,"data_stored":1086431232,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724591,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576704,"kb_used_data":237040,"kb_used_omap":4,"kb_used_meta":339643,"kb_avail":93795136,"statfs":{"total":96636764160,"available":96046219264,"internally_reserved":0,"allocated":242728960,"data_stored":242744095,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":347795151},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397538,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":274864,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":281460736,"data_stored":281172255,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234015,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247914,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450960,"kb_used_data":274880,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920880,"statfs":{"total":96636764160,"available":96174981120,"internally_reserved":0,"allocated":281477120,"data_stored":281495839,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":243486720,"data_stored":243490816,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:11.252 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:11.391 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:11.391 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:11.401 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2026-02-22T23:34:10.184121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9751,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127929,"num_bytes_recovered":1048281884,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1342096,"kb_used_data":786784,"kb_used_omap":14,"kb_used_meta":555249,"kb_avail":281773424,"statfs":{"total":289910292480,"available":288535986176,"internally_reserved":0,"allocated":805666816,"data_stored":805412189,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":568575446},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-7895,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7895,"num_bytes_recovered":64675840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001389"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":324648,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:34:05.539007+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:34:05.539007+0000","last_peered":"2026-02-22T23:34:05.539007+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:34:05.539007+0000","last_undegraded":"2026-02-22T23:34:05.539007+0000","last_fullsized":"2026-02-22T23:34:05.539007+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9751,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127927,"num_bytes_recovered":1047954204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9751,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127927,"num_bytes_recovered":1047954204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1086414848,"data_stored":1086431232,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724591,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576704,"kb_used_data":237040,"kb_used_omap":4,"kb_used_meta":339643,"kb_avail":93795136,"statfs":{"total":96636764160,"available":96046219264,"internally_reserved":0,"allocated":242728960,"data_stored":242744095,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":347795151},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397538,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":274864,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":281460736,"data_stored":281172255,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234015,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247914,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450960,"kb_used_data":274880,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920880,"statfs":{"total":96636764160,"available":96174981120,"internally_reserved":0,"allocated":281477120,"data_stored":281495839,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":243486720,"data_stored":243490816,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:14.403 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:14.555 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:14.555 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:14.567 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2026-02-22T23:34:14.184620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5866,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131814,"num_bytes_recovered":1080107804,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1353804,"kb_used_data":802652,"kb_used_omap":14,"kb_used_meta":551089,"kb_avail":281761716,"statfs":{"total":289910292480,"available":288523997184,"internally_reserved":0,"allocated":821915648,"data_stored":821661021,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":564315606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-7913,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7913,"num_bytes_recovered":64823296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001444"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":332418,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:34:10.539473+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:34:10.539473+0000","last_peered":"2026-02-22T23:34:10.539473+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:34:10.539473+0000","last_undegraded":"2026-02-22T23:34:10.539473+0000","last_fullsized":"2026-02-22T23:34:10.539473+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5866,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131812,"num_bytes_recovered":1079780124,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5866,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131812,"num_bytes_recovered":1079780124,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1102213120,"data_stored":1102229504,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724592,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588412,"kb_used_data":252908,"kb_used_omap":4,"kb_used_meta":335483,"kb_avail":93783428,"statfs":{"total":96636764160,"available":96034230272,"internally_reserved":0,"allocated":258977792,"data_stored":258992927,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":343535311},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397539,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":274864,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":281460736,"data_stored":281172255,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247915,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450960,"kb_used_data":274880,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920880,"statfs":{"total":96636764160,"available":96174981120,"internally_reserved":0,"allocated":281477120,"data_stored":281495839,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":259284992,"data_stored":259289088,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:14.568 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:14.714 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:14.714 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:14.725 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2026-02-22T23:34:14.184620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5866,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131814,"num_bytes_recovered":1080107804,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1353804,"kb_used_data":802652,"kb_used_omap":14,"kb_used_meta":551089,"kb_avail":281761716,"statfs":{"total":289910292480,"available":288523997184,"internally_reserved":0,"allocated":821915648,"data_stored":821661021,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":564315606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-7913,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7913,"num_bytes_recovered":64823296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001444"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":332418,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:34:10.539473+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:34:10.539473+0000","last_peered":"2026-02-22T23:34:10.539473+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:34:10.539473+0000","last_undegraded":"2026-02-22T23:34:10.539473+0000","last_fullsized":"2026-02-22T23:34:10.539473+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5866,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131812,"num_bytes_recovered":1079780124,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5866,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131812,"num_bytes_recovered":1079780124,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1102213120,"data_stored":1102229504,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724592,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588412,"kb_used_data":252908,"kb_used_omap":4,"kb_used_meta":335483,"kb_avail":93783428,"statfs":{"total":96636764160,"available":96034230272,"internally_reserved":0,"allocated":258977792,"data_stored":258992927,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":343535311},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397539,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":274864,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":281460736,"data_stored":281172255,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247915,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450960,"kb_used_data":274880,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920880,"statfs":{"total":96636764160,"available":96174981120,"internally_reserved":0,"allocated":281477120,"data_stored":281495839,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":259284992,"data_stored":259289088,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:17.727 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:17.872 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:17.872 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:17.883 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2026-02-22T23:34:16.184880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1979,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135700,"num_bytes_recovered":1111941916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1353804,"kb_used_data":802652,"kb_used_omap":14,"kb_used_meta":551089,"kb_avail":281761716,"statfs":{"total":289910292480,"available":288523997184,"internally_reserved":0,"allocated":821915648,"data_stored":821661021,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":564315606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-11800,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11799,"num_bytes_recovered":96657408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001488"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":340191,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:34:15.539674+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:34:15.539674+0000","last_peered":"2026-02-22T23:34:15.539674+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:34:15.539674+0000","last_undegraded":"2026-02-22T23:34:15.539674+0000","last_fullsized":"2026-02-22T23:34:15.539674+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1979,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135698,"num_bytes_recovered":1111614236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1979,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135698,"num_bytes_recovered":1111614236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1102213120,"data_stored":1102229504,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724592,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588412,"kb_used_data":252908,"kb_used_omap":4,"kb_used_meta":335483,"kb_avail":93783428,"statfs":{"total":96636764160,"available":96034230272,"internally_reserved":0,"allocated":258977792,"data_stored":258992927,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":343535311},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397540,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":274864,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":281460736,"data_stored":281172255,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247915,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450960,"kb_used_data":274880,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920880,"statfs":{"total":96636764160,"available":96174981120,"internally_reserved":0,"allocated":281477120,"data_stored":281495839,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":259284992,"data_stored":259289088,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:17.884 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:18.031 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:18.032 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:18.044 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2026-02-22T23:34:16.184880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1979,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135700,"num_bytes_recovered":1111941916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1353804,"kb_used_data":802652,"kb_used_omap":14,"kb_used_meta":551089,"kb_avail":281761716,"statfs":{"total":289910292480,"available":288523997184,"internally_reserved":0,"allocated":821915648,"data_stored":821661021,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":564315606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-11800,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11799,"num_bytes_recovered":96657408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001488"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":340191,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:34:15.539674+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:34:15.539674+0000","last_peered":"2026-02-22T23:34:15.539674+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:34:15.539674+0000","last_undegraded":"2026-02-22T23:34:15.539674+0000","last_fullsized":"2026-02-22T23:34:15.539674+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1979,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135698,"num_bytes_recovered":1111614236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":58,"state":"active+clean","last_fresh":"2026-02-22T23:32:41.662643+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:32:41.662643+0000","last_peered":"2026-02-22T23:32:41.662643+0000","last_clean":"2026-02-22T23:32:41.662643+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:32:41.662643+0000","last_undegraded":"2026-02-22T23:32:41.662643+0000","last_fullsized":"2026-02-22T23:32:41.662643+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1979,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135698,"num_bytes_recovered":1111614236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1102213120,"data_stored":1102229504,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724592,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588412,"kb_used_data":252908,"kb_used_omap":4,"kb_used_meta":335483,"kb_avail":93783428,"statfs":{"total":96636764160,"available":96034230272,"internally_reserved":0,"allocated":258977792,"data_stored":258992927,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":343535311},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397540,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":274864,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":281460736,"data_stored":281172255,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247915,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450960,"kb_used_data":274880,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920880,"statfs":{"total":96636764160,"available":96174981120,"internally_reserved":0,"allocated":281477120,"data_stored":281495839,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":259284992,"data_stored":259289088,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:21.045 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:21.206 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:21.207 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:21.219 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2026-02-22T23:34:20.185395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1979,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135700,"num_bytes_recovered":1111941916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1405532,"kb_used_data":818284,"kb_used_omap":14,"kb_used_meta":587185,"kb_avail":281709988,"statfs":{"total":289910292480,"available":288471027712,"internally_reserved":0,"allocated":837922816,"data_stored":837668189,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":601277910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3887,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3886,"num_bytes_recovered":31834112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001034"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":340191,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:34:15.539674+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:34:15.539674+0000","last_peered":"2026-02-22T23:34:15.539674+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:34:15.539674+0000","last_undegraded":"2026-02-22T23:34:15.539674+0000","last_fullsized":"2026-02-22T23:34:15.539674+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1979,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135698,"num_bytes_recovered":1111614236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":51,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-22T23:34:18.136809+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:34:18.136809+0000","last_peered":"2026-02-22T23:34:18.136809+0000","last_clean":"2026-02-22T23:34:18.136809+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:34:18.136809+0000","last_undegraded":"2026-02-22T23:34:18.136809+0000","last_fullsized":"2026-02-22T23:34:18.136809+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1979,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135698,"num_bytes_recovered":1111614236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1118003200,"data_stored":1118019584,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724593,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640140,"kb_used_data":268540,"kb_used_omap":4,"kb_used_meta":371579,"kb_avail":93731700,"statfs":{"total":96636764160,"available":95981260800,"internally_reserved":0,"allocated":274984960,"data_stored":275000095,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":380497615},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397540,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":274864,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":281460736,"data_stored":281172255,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234017,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247916,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450960,"kb_used_data":274880,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920880,"statfs":{"total":96636764160,"available":96174981120,"internally_reserved":0,"allocated":281477120,"data_stored":281495839,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":275075072,"data_stored":275079168,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:21.220 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:21.382 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:21.382 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:21.393 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2026-02-22T23:34:20.185395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1979,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135700,"num_bytes_recovered":1111941916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1405532,"kb_used_data":818284,"kb_used_omap":14,"kb_used_meta":587185,"kb_avail":281709988,"statfs":{"total":289910292480,"available":288471027712,"internally_reserved":0,"allocated":837922816,"data_stored":837668189,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":601277910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3887,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3886,"num_bytes_recovered":31834112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001034"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":340191,"reported_epoch":58,"state":"active+remapped+backfilling","last_fresh":"2026-02-22T23:34:15.539674+0000","last_change":"2026-02-22T23:33:49.472877+0000","last_active":"2026-02-22T23:34:15.539674+0000","last_peered":"2026-02-22T23:34:15.539674+0000","last_clean":"2026-02-22T23:32:35.643295+0000","last_became_active":"2026-02-22T23:32:41.677538+0000","last_became_peered":"2026-02-22T23:32:41.677538+0000","last_unstale":"2026-02-22T23:34:15.539674+0000","last_undegraded":"2026-02-22T23:34:15.539674+0000","last_fullsized":"2026-02-22T23:34:15.539674+0000","mapping_epoch":57,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1979,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135698,"num_bytes_recovered":1111614236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":68599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":51,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-22T23:34:18.136809+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:34:18.136809+0000","last_peered":"2026-02-22T23:34:18.136809+0000","last_clean":"2026-02-22T23:34:18.136809+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:34:18.136809+0000","last_undegraded":"2026-02-22T23:34:18.136809+0000","last_fullsized":"2026-02-22T23:34:18.136809+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1979,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135698,"num_bytes_recovered":1111614236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1118003200,"data_stored":1118019584,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724593,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640140,"kb_used_data":268540,"kb_used_omap":4,"kb_used_meta":371579,"kb_avail":93731700,"statfs":{"total":96636764160,"available":95981260800,"internally_reserved":0,"allocated":274984960,"data_stored":275000095,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":380497615},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397540,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314432,"kb_used_data":274864,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057408,"statfs":{"total":96636764160,"available":96314785792,"internally_reserved":0,"allocated":281460736,"data_stored":281172255,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234017,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247916,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450960,"kb_used_data":274880,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920880,"statfs":{"total":96636764160,"available":96174981120,"internally_reserved":0,"allocated":281477120,"data_stored":281495839,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":275075072,"data_stored":275079168,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:24.395 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:24.550 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:24.550 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:24.560 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2026-02-22T23:34:24.185872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137679,"num_bytes_recovered":1128153884,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1407784,"kb_used_data":824680,"kb_used_omap":14,"kb_used_meta":583025,"kb_avail":281707736,"statfs":{"total":289910292480,"available":288468721664,"internally_reserved":0,"allocated":844472320,"data_stored":844189238,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":597018070},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1979,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":16211968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000992"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":344325,"reported_epoch":60,"state":"active+clean+scrubbing","last_fresh":"2026-02-22T23:34:20.529466+0000","last_change":"2026-02-22T23:34:19.344477+0000","last_active":"2026-02-22T23:34:20.529466+0000","last_peered":"2026-02-22T23:34:20.529466+0000","last_clean":"2026-02-22T23:34:20.529466+0000","last_became_active":"2026-02-22T23:34:19.142816+0000","last_became_peered":"2026-02-22T23:34:19.142816+0000","last_unstale":"2026-02-22T23:34:20.529466+0000","last_undegraded":"2026-02-22T23:34:20.529466+0000","last_fullsized":"2026-02-22T23:34:20.529466+0000","mapping_epoch":59,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":15999,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 2s","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137677,"num_bytes_recovered":1127826204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":53,"reported_epoch":60,"state":"active+clean","last_fresh":"2026-02-22T23:34:19.142084+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:34:19.142084+0000","last_peered":"2026-02-22T23:34:19.142084+0000","last_clean":"2026-02-22T23:34:19.142084+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:34:19.142084+0000","last_undegraded":"2026-02-22T23:34:19.142084+0000","last_fullsized":"2026-02-22T23:34:19.142084+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137677,"num_bytes_recovered":1127826204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123295232,"data_stored":1123311616,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642348,"kb_used_data":274904,"kb_used_omap":4,"kb_used_meta":367419,"kb_avail":93729492,"statfs":{"total":96636764160,"available":95978999808,"internally_reserved":0,"allocated":281501696,"data_stored":281507346,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":376237775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314452,"kb_used_data":274880,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057388,"statfs":{"total":96636764160,"available":96314765312,"internally_reserved":0,"allocated":281477120,"data_stored":281179154,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450984,"kb_used_data":274896,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920856,"statfs":{"total":96636764160,"available":96174956544,"internally_reserved":0,"allocated":281493504,"data_stored":281502738,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280367104,"data_stored":280371200,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:24.561 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:recovered! 2026-02-22T23:34:24.561 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-22T23:34:24.577 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-22T23:34:24.577 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:24.718 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:24.719 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:24.730 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2026-02-22T23:34:24.185872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137679,"num_bytes_recovered":1128153884,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1407784,"kb_used_data":824680,"kb_used_omap":14,"kb_used_meta":583025,"kb_avail":281707736,"statfs":{"total":289910292480,"available":288468721664,"internally_reserved":0,"allocated":844472320,"data_stored":844189238,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":597018070},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1979,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":16211968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000992"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":344325,"reported_epoch":60,"state":"active+clean+scrubbing","last_fresh":"2026-02-22T23:34:20.529466+0000","last_change":"2026-02-22T23:34:19.344477+0000","last_active":"2026-02-22T23:34:20.529466+0000","last_peered":"2026-02-22T23:34:20.529466+0000","last_clean":"2026-02-22T23:34:20.529466+0000","last_became_active":"2026-02-22T23:34:19.142816+0000","last_became_peered":"2026-02-22T23:34:19.142816+0000","last_unstale":"2026-02-22T23:34:20.529466+0000","last_undegraded":"2026-02-22T23:34:20.529466+0000","last_fullsized":"2026-02-22T23:34:20.529466+0000","mapping_epoch":59,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":15999,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 2s","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137677,"num_bytes_recovered":1127826204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":53,"reported_epoch":60,"state":"active+clean","last_fresh":"2026-02-22T23:34:19.142084+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:34:19.142084+0000","last_peered":"2026-02-22T23:34:19.142084+0000","last_clean":"2026-02-22T23:34:19.142084+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:34:19.142084+0000","last_undegraded":"2026-02-22T23:34:19.142084+0000","last_fullsized":"2026-02-22T23:34:19.142084+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137677,"num_bytes_recovered":1127826204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123295232,"data_stored":1123311616,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642348,"kb_used_data":274904,"kb_used_omap":4,"kb_used_meta":367419,"kb_avail":93729492,"statfs":{"total":96636764160,"available":95978999808,"internally_reserved":0,"allocated":281501696,"data_stored":281507346,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":376237775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314452,"kb_used_data":274880,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057388,"statfs":{"total":96636764160,"available":96314765312,"internally_reserved":0,"allocated":281477120,"data_stored":281179154,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450984,"kb_used_data":274896,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920856,"statfs":{"total":96636764160,"available":96174956544,"internally_reserved":0,"allocated":281493504,"data_stored":281502738,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280367104,"data_stored":280371200,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:24.731 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:24.884 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:24.884 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:24.896 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2026-02-22T23:34:24.185872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137679,"num_bytes_recovered":1128153884,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1407784,"kb_used_data":824680,"kb_used_omap":14,"kb_used_meta":583025,"kb_avail":281707736,"statfs":{"total":289910292480,"available":288468721664,"internally_reserved":0,"allocated":844472320,"data_stored":844189238,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":597018070},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1979,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":16211968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000992"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":344325,"reported_epoch":60,"state":"active+clean+scrubbing","last_fresh":"2026-02-22T23:34:20.529466+0000","last_change":"2026-02-22T23:34:19.344477+0000","last_active":"2026-02-22T23:34:20.529466+0000","last_peered":"2026-02-22T23:34:20.529466+0000","last_clean":"2026-02-22T23:34:20.529466+0000","last_became_active":"2026-02-22T23:34:19.142816+0000","last_became_peered":"2026-02-22T23:34:19.142816+0000","last_unstale":"2026-02-22T23:34:20.529466+0000","last_undegraded":"2026-02-22T23:34:20.529466+0000","last_fullsized":"2026-02-22T23:34:20.529466+0000","mapping_epoch":59,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":15999,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 2s","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137677,"num_bytes_recovered":1127826204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":53,"reported_epoch":60,"state":"active+clean","last_fresh":"2026-02-22T23:34:19.142084+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:34:19.142084+0000","last_peered":"2026-02-22T23:34:19.142084+0000","last_clean":"2026-02-22T23:34:19.142084+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:34:19.142084+0000","last_undegraded":"2026-02-22T23:34:19.142084+0000","last_fullsized":"2026-02-22T23:34:19.142084+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137677,"num_bytes_recovered":1127826204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123295232,"data_stored":1123311616,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642348,"kb_used_data":274904,"kb_used_omap":4,"kb_used_meta":367419,"kb_avail":93729492,"statfs":{"total":96636764160,"available":95978999808,"internally_reserved":0,"allocated":281501696,"data_stored":281507346,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":376237775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314452,"kb_used_data":274880,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057388,"statfs":{"total":96636764160,"available":96314765312,"internally_reserved":0,"allocated":281477120,"data_stored":281179154,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450984,"kb_used_data":274896,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920856,"statfs":{"total":96636764160,"available":96174956544,"internally_reserved":0,"allocated":281493504,"data_stored":281502738,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280367104,"data_stored":280371200,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:24.897 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-22T23:34:24.897 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:25.043 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:25.043 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:25.055 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2026-02-22T23:34:24.185872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562283406,"num_objects":68601,"num_object_clones":0,"num_object_copies":205801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68601,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":68636,"num_write_kb":549109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137679,"num_bytes_recovered":1128153884,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1407784,"kb_used_data":824680,"kb_used_omap":14,"kb_used_meta":583025,"kb_avail":281707736,"statfs":{"total":289910292480,"available":288468721664,"internally_reserved":0,"allocated":844472320,"data_stored":844189238,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":597018070},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1979,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1979,"num_bytes_recovered":16211968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000992"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":344325,"reported_epoch":60,"state":"active+clean+scrubbing","last_fresh":"2026-02-22T23:34:20.529466+0000","last_change":"2026-02-22T23:34:19.344477+0000","last_active":"2026-02-22T23:34:20.529466+0000","last_peered":"2026-02-22T23:34:20.529466+0000","last_clean":"2026-02-22T23:34:20.529466+0000","last_became_active":"2026-02-22T23:34:19.142816+0000","last_became_peered":"2026-02-22T23:34:19.142816+0000","last_unstale":"2026-02-22T23:34:20.529466+0000","last_undegraded":"2026-02-22T23:34:20.529466+0000","last_fullsized":"2026-02-22T23:34:20.529466+0000","mapping_epoch":59,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:28:22.258373+0000","objects_scrubbed":15999,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 2s","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137677,"num_bytes_recovered":1127826204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":53,"reported_epoch":60,"state":"active+clean","last_fresh":"2026-02-22T23:34:19.142084+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:34:19.142084+0000","last_peered":"2026-02-22T23:34:19.142084+0000","last_clean":"2026-02-22T23:34:19.142084+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:34:19.142084+0000","last_undegraded":"2026-02-22T23:34:19.142084+0000","last_fullsized":"2026-02-22T23:34:19.142084+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561955214,"num_objects":68599,"num_object_clones":0,"num_object_copies":205797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68599,"num_write_kb":548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137677,"num_bytes_recovered":1127826204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123295232,"data_stored":1123311616,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642348,"kb_used_data":274904,"kb_used_omap":4,"kb_used_meta":367419,"kb_avail":93729492,"statfs":{"total":96636764160,"available":95978999808,"internally_reserved":0,"allocated":281501696,"data_stored":281507346,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":376237775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314452,"kb_used_data":274880,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057388,"statfs":{"total":96636764160,"available":96314765312,"internally_reserved":0,"allocated":281477120,"data_stored":281179154,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450984,"kb_used_data":274896,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920856,"statfs":{"total":96636764160,"available":96174956544,"internally_reserved":0,"allocated":281493504,"data_stored":281502738,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280367104,"data_stored":280371200,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:25.056 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-22T23:34:25.214 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:25.214 INFO:teuthology.orchestra.run.trial115.stdout:{"epoch":60,"fsid":"a4a3ada6-bd8c-4e08-a984-e18b05a50ce6","created":"2026-02-22T23:27:24.074444+0000","modified":"2026-02-22T23:34:19.135715+0000","last_up_change":"2026-02-22T23:32:34.631263+0000","last_in_change":"2026-02-22T23:32:39.648005+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":11,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-22T23:27:34.171905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-22T23:27:43.179120+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"bffc6a3f-193a-457c-aedc-72aed6eb39e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":35,"up_from":39,"up_thru":39,"down_at":36,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6809","nonce":2954119946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6810","nonce":2954119946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6812","nonce":2954119946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6811","nonce":2954119946}]},"public_addr":"10.20.193.115:6809/2954119946","cluster_addr":"10.20.193.115:6810/2954119946","heartbeat_back_addr":"10.20.193.115:6812/2954119946","heartbeat_front_addr":"10.20.193.115:6811/2954119946","state":["exists","up"]},{"osd":1,"uuid":"5b969a7a-bb01-4c32-84bc-ca4ac4d91e33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":23,"up_thru":46,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6805","nonce":2409177103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6806","nonce":2409177103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6808","nonce":2409177103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6807","nonce":2409177103}]},"public_addr":"10.20.193.115:6805/2409177103","cluster_addr":"10.20.193.115:6806/2409177103","heartbeat_back_addr":"10.20.193.115:6808/2409177103","heartbeat_front_addr":"10.20.193.115:6807/2409177103","state":["exists","up"]},{"osd":2,"uuid":"6da393ab-bd56-4d92-a2e8-9807271ac6e4","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":36,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6801","nonce":237665470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6802","nonce":237665470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6804","nonce":237665470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6803","nonce":237665470}]},"public_addr":"10.20.193.115:6801/237665470","cluster_addr":"10.20.193.115:6802/237665470","heartbeat_back_addr":"10.20.193.115:6804/237665470","heartbeat_front_addr":"10.20.193.115:6803/237665470","state":["exists","up"]},{"osd":3,"uuid":"288ba70e-ec1b-4b0e-8609-cafdcbba9bda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":29,"last_clean_end":45,"up_from":49,"up_thru":59,"down_at":46,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6813","nonce":428774140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6814","nonce":428774140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6816","nonce":428774140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.115:6815","nonce":428774140}]},"public_addr":"10.20.193.115:6813/428774140","cluster_addr":"10.20.193.115:6814/428774140","heartbeat_back_addr":"10.20.193.115:6816/428774140","heartbeat_front_addr":"10.20.193.115:6815/428774140","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:32.314887+0000","dead_epoch":36},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:31.613759+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2026-02-22T23:27:31.523737+0000","dead_epoch":51},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-22T23:27:32.017476+0000","dead_epoch":46}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-22T23:34:26.171 INFO:tasks.ceph.mgr.x.trial115.stderr:2026-02-22T23:34:26.168+0000 7f276fb06640 -1 calc_pg_upmaps abort due to max <= 0 2026-02-22T23:34:26.226 INFO:tasks.ceph:Scrubbing osd.0 2026-02-22T23:34:26.226 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-22T23:34:26.308 INFO:teuthology.orchestra.run.trial115.stdout:{ 2026-02-22T23:34:26.308 INFO:teuthology.orchestra.run.trial115.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T23:34:26.308 INFO:teuthology.orchestra.run.trial115.stdout:} 2026-02-22T23:34:26.317 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-22T23:34:26.468 INFO:teuthology.orchestra.run.trial115.stderr:instructed osd(s) 0 to deep-scrub 2026-02-22T23:34:26.478 INFO:tasks.ceph:Scrubbing osd.1 2026-02-22T23:34:26.478 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-22T23:34:26.555 INFO:teuthology.orchestra.run.trial115.stdout:{ 2026-02-22T23:34:26.555 INFO:teuthology.orchestra.run.trial115.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T23:34:26.555 INFO:teuthology.orchestra.run.trial115.stdout:} 2026-02-22T23:34:26.564 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-22T23:34:26.708 INFO:teuthology.orchestra.run.trial115.stderr:instructed osd(s) 1 to deep-scrub 2026-02-22T23:34:26.719 INFO:tasks.ceph:Scrubbing osd.2 2026-02-22T23:34:26.719 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-22T23:34:26.794 INFO:teuthology.orchestra.run.trial115.stdout:{ 2026-02-22T23:34:26.794 INFO:teuthology.orchestra.run.trial115.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T23:34:26.795 INFO:teuthology.orchestra.run.trial115.stdout:} 2026-02-22T23:34:26.803 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-22T23:34:26.949 INFO:teuthology.orchestra.run.trial115.stderr:instructed osd(s) 2 to deep-scrub 2026-02-22T23:34:26.961 INFO:tasks.ceph:Scrubbing osd.3 2026-02-22T23:34:26.961 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2026-02-22T23:34:27.043 INFO:teuthology.orchestra.run.trial115.stdout:{ 2026-02-22T23:34:27.043 INFO:teuthology.orchestra.run.trial115.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-22T23:34:27.043 INFO:teuthology.orchestra.run.trial115.stdout:} 2026-02-22T23:34:27.052 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2026-02-22T23:34:27.203 INFO:teuthology.orchestra.run.trial115.stderr:instructed osd(s) 3 to deep-scrub 2026-02-22T23:34:27.214 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:27.365 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:27.366 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:27.376 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2026-02-22T23:34:26.186115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562275027,"num_objects":68600,"num_object_clones":0,"num_object_copies":205798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68600,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1407784,"kb_used_data":824680,"kb_used_omap":14,"kb_used_meta":583025,"kb_avail":281707736,"statfs":{"total":289910292480,"available":288468721664,"internally_reserved":0,"allocated":844472320,"data_stored":844189238,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":597018070},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8379,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1979,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-68599,"num_write_kb":-548787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-135698,"num_bytes_recovered":-1111614236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":-1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.050009"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":344855,"reported_epoch":60,"state":"active+clean","last_fresh":"2026-02-22T23:34:24.176879+0000","last_change":"2026-02-22T23:34:24.176914+0000","last_active":"2026-02-22T23:34:24.176879+0000","last_peered":"2026-02-22T23:34:24.176879+0000","last_clean":"2026-02-22T23:34:24.176879+0000","last_became_active":"2026-02-22T23:34:19.142816+0000","last_became_peered":"2026-02-22T23:34:19.142816+0000","last_unstale":"2026-02-22T23:34:24.176879+0000","last_undegraded":"2026-02-22T23:34:24.176879+0000","last_fullsized":"2026-02-22T23:34:24.176879+0000","mapping_epoch":59,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'68600","last_scrub_stamp":"2026-02-22T23:34:24.176825+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-22T23:28:22.258373+0000","last_clean_scrub_stamp":"2026-02-22T23:34:24.176825+0000","objects_scrubbed":68598,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:19:34.902581+0000","scrub_duration":4831,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561946835,"num_objects":68598,"num_object_clones":0,"num_object_copies":205794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":53,"reported_epoch":60,"state":"active+clean","last_fresh":"2026-02-22T23:34:19.142084+0000","last_change":"2026-02-22T23:31:37.441871+0000","last_active":"2026-02-22T23:34:19.142084+0000","last_peered":"2026-02-22T23:34:19.142084+0000","last_clean":"2026-02-22T23:34:19.142084+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:34:19.142084+0000","last_undegraded":"2026-02-22T23:34:19.142084+0000","last_fullsized":"2026-02-22T23:34:19.142084+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-22T23:27:35.152922+0000","last_clean_scrub_stamp":"2026-02-22T23:27:35.152922+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T03:52:38.076206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561946835,"num_objects":68598,"num_object_clones":0,"num_object_copies":205794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123295232,"data_stored":1123311616,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642348,"kb_used_data":274904,"kb_used_omap":4,"kb_used_meta":367419,"kb_avail":93729492,"statfs":{"total":96636764160,"available":95978999808,"internally_reserved":0,"allocated":281501696,"data_stored":281507346,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":376237775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397542,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314452,"kb_used_data":274880,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057388,"statfs":{"total":96636764160,"available":96314765312,"internally_reserved":0,"allocated":281477120,"data_stored":281179154,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450984,"kb_used_data":274896,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920856,"statfs":{"total":96636764160,"available":96174956544,"internally_reserved":0,"allocated":281493504,"data_stored":281502738,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280367104,"data_stored":280371200,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:27.381 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2026-02-22T23:34:24.176825+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=23, tm_min=34, tm_sec=24, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=23, tm_min=34, tm_sec=25, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T23:34:27.381 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-22T23:27:35.152922+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=23, tm_min=27, tm_sec=35, tm_wday=6, tm_yday=53, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=22, tm_hour=23, tm_min=34, tm_sec=25, tm_wday=6, tm_yday=53, tm_isdst=0) 2026-02-22T23:34:27.381 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-22T23:34:47.382 DEBUG:teuthology.orchestra.run.trial115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-22T23:34:47.540 INFO:teuthology.orchestra.run.trial115.stdout: 2026-02-22T23:34:47.540 INFO:teuthology.orchestra.run.trial115.stderr:dumped all 2026-02-22T23:34:47.549 INFO:teuthology.orchestra.run.trial115.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2026-02-22T23:34:46.188708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562275027,"num_objects":68600,"num_object_clones":0,"num_object_copies":205798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68600,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48,"ondisk_log_size":48,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1407768,"kb_used_data":824680,"kb_used_omap":14,"kb_used_meta":583025,"kb_avail":281707752,"statfs":{"total":289910292480,"available":288468738048,"internally_reserved":0,"allocated":844472320,"data_stored":844189238,"data_compressed":9240,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14890,"internal_metadata":597018070},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001630"},"pg_stats":[{"pgid":"2.0","version":"58'68600","reported_seq":349438,"reported_epoch":60,"state":"active+clean","last_fresh":"2026-02-22T23:34:43.458005+0000","last_change":"2026-02-22T23:34:43.458027+0000","last_active":"2026-02-22T23:34:43.458005+0000","last_peered":"2026-02-22T23:34:43.458005+0000","last_clean":"2026-02-22T23:34:43.458005+0000","last_became_active":"2026-02-22T23:34:19.142816+0000","last_became_peered":"2026-02-22T23:34:19.142816+0000","last_unstale":"2026-02-22T23:34:43.458005+0000","last_undegraded":"2026-02-22T23:34:43.458005+0000","last_fullsized":"2026-02-22T23:34:43.458005+0000","mapping_epoch":59,"log_start":"25'68571","ondisk_log_start":"25'68571","created":16,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'68600","last_scrub_stamp":"2026-02-22T23:34:43.457889+0000","last_deep_scrub":"58'68600","last_deep_scrub_stamp":"2026-02-22T23:34:43.457889+0000","last_clean_scrub_stamp":"2026-02-22T23:34:43.457889+0000","objects_scrubbed":68598,"log_size":29,"log_dups_size":3000,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":17,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T07:56:26.905444+0000","scrub_duration":16231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561946835,"num_objects":68598,"num_object_clones":0,"num_object_copies":205794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"60'19","reported_seq":64,"reported_epoch":60,"state":"active+clean","last_fresh":"2026-02-22T23:34:26.886774+0000","last_change":"2026-02-22T23:34:26.886805+0000","last_active":"2026-02-22T23:34:26.886774+0000","last_peered":"2026-02-22T23:34:26.886774+0000","last_clean":"2026-02-22T23:34:26.886774+0000","last_became_active":"2026-02-22T23:31:37.437734+0000","last_became_peered":"2026-02-22T23:31:37.437734+0000","last_unstale":"2026-02-22T23:34:26.886774+0000","last_undegraded":"2026-02-22T23:34:26.886774+0000","last_fullsized":"2026-02-22T23:34:26.886774+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'19","last_scrub_stamp":"2026-02-22T23:34:26.886724+0000","last_deep_scrub":"60'19","last_deep_scrub_stamp":"2026-02-22T23:34:26.886724+0000","last_clean_scrub_stamp":"2026-02-22T23:34:26.886724+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-24T10:17:53.130287+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":561946835,"num_objects":68598,"num_object_clones":0,"num_object_copies":205794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1120837632,"data_stored":1120854016,"data_compressed":11521,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1695,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":0,"up_from":39,"seq":167503724598,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642344,"kb_used_data":274904,"kb_used_omap":4,"kb_used_meta":367419,"kb_avail":93729496,"statfs":{"total":96636764160,"available":95979003904,"internally_reserved":0,"allocated":281501696,"data_stored":281507346,"data_compressed":4060,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4401,"internal_metadata":376237775},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":49,"seq":210453397546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314448,"kb_used_data":274880,"kb_used_omap":5,"kb_used_meta":39546,"kb_avail":94057392,"statfs":{"total":96636764160,"available":96314769408,"internally_reserved":0,"allocated":281477120,"data_stored":281179154,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6092,"internal_metadata":40495156},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":231928234022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247921,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450976,"kb_used_data":274896,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93920864,"statfs":{"total":96636764160,"available":96174964736,"internally_reserved":0,"allocated":281493504,"data_stored":281502738,"data_compressed":2724,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4397,"internal_metadata":180285139},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280973312,"data_stored":280977408,"data_compressed":2131,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":277909504,"data_stored":277913600,"data_compressed":3467,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280977408,"data_stored":280981504,"data_compressed":2456,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1695,"internal_metadata":0}]}} 2026-02-22T23:34:47.551 DEBUG:teuthology.orchestra.run.trial115:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-22T23:34:47.710 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-22T23:34:47.710 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-22T23:34:47.711 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-22T23:34:47.711 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T23:34:47.801 INFO:tasks.ceph.osd.0:Stopped 2026-02-22T23:34:47.801 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-22T23:34:47.801 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T23:34:47.897 INFO:tasks.ceph.osd.1:Stopped 2026-02-22T23:34:47.897 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-22T23:34:47.898 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T23:34:53.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2026-02-22T23:34:53.999 INFO:tasks.ceph.osd.2:Stopped 2026-02-22T23:34:53.999 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-02-22T23:34:53.999 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T23:34:54.073 INFO:tasks.ceph.osd.3:Stopped 2026-02-22T23:34:54.073 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-22T23:34:54.074 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-22T23:34:54.074 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T23:34:54.101 INFO:tasks.ceph.mgr.x:Stopped 2026-02-22T23:34:54.101 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-22T23:34:54.101 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-22T23:34:54.101 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T23:34:54.154 INFO:tasks.ceph.mon.a:Stopped 2026-02-22T23:34:54.154 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2026-02-22T23:34:54.154 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T23:34:54.206 INFO:tasks.ceph.mon.b:Stopped 2026-02-22T23:34:54.206 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2026-02-22T23:34:54.206 INFO:teuthology.orchestra.run:waiting for 300 2026-02-22T23:34:54.263 INFO:tasks.ceph.mon.c:Stopped 2026-02-22T23:34:54.263 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-22T23:34:54.263 DEBUG:teuthology.orchestra.run.trial115:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(OBJECT_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(SLOW_OPS\)' | egrep -v deep-scrub | egrep -v missing | egrep -v 'overall HEALTH_' | egrep -v repair | egrep -v 'slow request' | egrep -v unfound | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'enough copies available' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2026-02-22T23:34:54.317 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial115.front.sepia.ceph.com 2026-02-22T23:34:54.318 DEBUG:teuthology.orchestra.run.trial115:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-22T23:34:55.798 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial115.front.sepia.ceph.com 2026-02-22T23:34:55.798 DEBUG:teuthology.orchestra.run.trial115:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-22T23:34:55.821 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial115.front.sepia.ceph.com 2026-02-22T23:34:55.822 DEBUG:teuthology.orchestra.run.trial115:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-22T23:34:55.882 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@trial115.front.sepia.ceph.com 2026-02-22T23:34:55.883 DEBUG:teuthology.orchestra.run.trial115:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2026-02-22T23:34:55.941 INFO:tasks.ceph:Archiving mon data... 2026-02-22T23:34:55.943 DEBUG:teuthology.misc:Transferring archived files from trial115:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64402/data/mon.a.tgz 2026-02-22T23:34:55.943 DEBUG:teuthology.orchestra.run.trial115:> mktemp 2026-02-22T23:34:55.986 INFO:teuthology.orchestra.run.trial115.stdout:/tmp/tmp.YIHDcLteFI 2026-02-22T23:34:55.986 DEBUG:teuthology.orchestra.run.trial115:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.YIHDcLteFI 2026-02-22T23:34:56.070 DEBUG:teuthology.orchestra.run.trial115:> sudo chmod 0666 /tmp/tmp.YIHDcLteFI 2026-02-22T23:34:56.168 DEBUG:teuthology.orchestra.remote:trial115:/tmp/tmp.YIHDcLteFI is 291KB 2026-02-22T23:34:56.224 DEBUG:teuthology.orchestra.run.trial115:> rm -fr /tmp/tmp.YIHDcLteFI 2026-02-22T23:34:56.228 DEBUG:teuthology.misc:Transferring archived files from trial115:/var/lib/ceph/mon/ceph-b to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64402/data/mon.b.tgz 2026-02-22T23:34:56.229 DEBUG:teuthology.orchestra.run.trial115:> mktemp 2026-02-22T23:34:56.274 INFO:teuthology.orchestra.run.trial115.stdout:/tmp/tmp.seRjYxlmOm 2026-02-22T23:34:56.274 DEBUG:teuthology.orchestra.run.trial115:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.seRjYxlmOm 2026-02-22T23:34:56.377 DEBUG:teuthology.orchestra.run.trial115:> sudo chmod 0666 /tmp/tmp.seRjYxlmOm 2026-02-22T23:34:56.436 DEBUG:teuthology.orchestra.remote:trial115:/tmp/tmp.seRjYxlmOm is 377KB 2026-02-22T23:34:56.492 DEBUG:teuthology.orchestra.run.trial115:> rm -fr /tmp/tmp.seRjYxlmOm 2026-02-22T23:34:56.496 DEBUG:teuthology.misc:Transferring archived files from trial115:/var/lib/ceph/mon/ceph-c to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64402/data/mon.c.tgz 2026-02-22T23:34:56.497 DEBUG:teuthology.orchestra.run.trial115:> mktemp 2026-02-22T23:34:56.542 INFO:teuthology.orchestra.run.trial115.stdout:/tmp/tmp.M9zFJrbhq4 2026-02-22T23:34:56.542 DEBUG:teuthology.orchestra.run.trial115:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.M9zFJrbhq4 2026-02-22T23:34:56.638 DEBUG:teuthology.orchestra.run.trial115:> sudo chmod 0666 /tmp/tmp.M9zFJrbhq4 2026-02-22T23:34:56.697 DEBUG:teuthology.orchestra.remote:trial115:/tmp/tmp.M9zFJrbhq4 is 376KB 2026-02-22T23:34:56.765 DEBUG:teuthology.orchestra.run.trial115:> rm -fr /tmp/tmp.M9zFJrbhq4 2026-02-22T23:34:56.770 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-22T23:34:56.770 DEBUG:teuthology.orchestra.run.trial115:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-22T23:34:56.862 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-22T23:34:56.862 INFO:tasks.ceph:Archiving crash dumps... 2026-02-22T23:34:56.865 DEBUG:teuthology.misc:Transferring archived files from trial115:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64402/remote/trial115/crash 2026-02-22T23:34:56.866 DEBUG:teuthology.orchestra.run.trial115:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-22T23:34:56.910 INFO:tasks.ceph:Compressing logs... 2026-02-22T23:34:56.911 DEBUG:teuthology.orchestra.run.trial115:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T23:34:56.963 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45648.log 2026-02-22T23:34:56.966 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27729.log 2026-02-22T23:34:56.966 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29042.log 2026-02-22T23:34:56.966 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45648.log.gz 2026-02-22T23:34:56.966 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27729.log.gz 2026-02-22T23:34:56.967 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24404.log 2026-02-22T23:34:56.967 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30868.log 2026-02-22T23:34:56.967 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29042.log: /var/log/ceph/ceph-client.admin.24404.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24404.log.gz 2026-02-22T23:34:56.967 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29042.log.gz 2026-02-22T23:34:56.967 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45412.log 2026-02-22T23:34:56.967 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40593.log 2026-02-22T23:34:56.967 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30868.log: /var/log/ceph/ceph-client.admin.45412.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30868.log.gz 2026-02-22T23:34:56.967 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45412.log.gz 2026-02-22T23:34:56.967 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25243.log 2026-02-22T23:34:56.967 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.40593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40593.log.gz 2026-02-22T23:34:56.967 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30427.log 2026-02-22T23:34:56.967 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29617.log 2026-02-22T23:34:56.968 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.25243.log: 0.0%/var/log/ceph/ceph-client.admin.30427.log: -- replaced with /var/log/ceph/ceph-client.admin.25243.log.gz 2026-02-22T23:34:56.968 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30427.log.gz 2026-02-22T23:34:56.968 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30641.log 2026-02-22T23:34:56.983 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44809.log 2026-02-22T23:34:56.984 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29617.log.gz 2026-02-22T23:34:56.984 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24908.log 2026-02-22T23:34:56.984 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30641.log.gz 2026-02-22T23:34:56.984 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40533.log 2026-02-22T23:34:56.984 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44809.log.gz 2026-02-22T23:34:56.984 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21871.log 2026-02-22T23:34:56.984 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24908.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24908.log.gz 2026-02-22T23:34:56.984 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.40533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42363.log 2026-02-22T23:34:56.984 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40533.log.gz 2026-02-22T23:34:56.984 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21871.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.21871.log.gz -- 2026-02-22T23:34:56.985 INFO:teuthology.orchestra.run.trial115.stderr: /var/log/ceph/ceph-client.admin.32302.log 2026-02-22T23:34:56.985 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42363.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42363.log.gz 2026-02-22T23:34:56.985 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2026-02-22T23:34:56.985 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21879.log 2026-02-22T23:34:56.985 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32302.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32302.log.gz 2026-02-22T23:34:56.985 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31515.log 2026-02-22T23:34:56.985 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32407.log 2026-02-22T23:34:56.985 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21879.log.gz 2026-02-22T23:34:56.985 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24534.log 2026-02-22T23:34:56.985 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31515.log.gz 2026-02-22T23:34:56.985 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35607.log 2026-02-22T23:34:56.985 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32407.log.gz 2026-02-22T23:34:56.986 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24534.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24534.log.gz 2026-02-22T23:34:56.986 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17515.log 2026-02-22T23:34:56.986 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24546.log 2026-02-22T23:34:56.986 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35607.log: /var/log/ceph/ceph-client.admin.17515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35607.log.gz 2026-02-22T23:34:56.986 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17515.log.gz 2026-02-22T23:34:56.986 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32870.log 2026-02-22T23:34:56.986 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42361.log 2026-02-22T23:34:56.986 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24546.log.gz 2026-02-22T23:34:56.986 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43923.log 2026-02-22T23:34:56.986 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32870.log.gz 2026-02-22T23:34:56.986 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42252.log 2026-02-22T23:34:56.986 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42361.log.gz 2026-02-22T23:34:56.987 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31720.log 2026-02-22T23:34:56.987 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43923.log.gz 2026-02-22T23:34:56.987 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42252.log.gzgzip 2026-02-22T23:34:56.987 INFO:teuthology.orchestra.run.trial115.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32504.log 2026-02-22T23:34:56.987 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28298.log 2026-02-22T23:34:56.987 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31720.log.gz 2026-02-22T23:34:56.987 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45960.log 2026-02-22T23:34:56.987 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32504.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32504.log.gz 2026-02-22T23:34:56.987 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36379.log 2026-02-22T23:34:56.987 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28298.log.gz 2026-02-22T23:34:56.987 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44797.log 2026-02-22T23:34:56.987 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45960.log.gz 2026-02-22T23:34:56.988 INFO:teuthology.orchestra.run.trial115.stderr:gzip/var/log/ceph/ceph-client.admin.36379.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.24441.log 2026-02-22T23:34:56.988 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36379.log.gz 2026-02-22T23:34:56.988 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32842.log 2026-02-22T23:34:56.988 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44797.log.gz 2026-02-22T23:34:56.988 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31188.log 2026-02-22T23:34:56.988 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24441.log.gz 2026-02-22T23:34:56.988 INFO:teuthology.orchestra.run.trial115.stderr:gzip/var/log/ceph/ceph-client.admin.32842.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28938.log 2026-02-22T23:34:56.988 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32842.log.gz 2026-02-22T23:34:56.988 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27284.log 2026-02-22T23:34:56.988 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31188.log.gz 2026-02-22T23:34:56.988 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28299.log 2026-02-22T23:34:56.988 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28938.log.gz 2026-02-22T23:34:56.988 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44153.log 2026-02-22T23:34:56.989 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27284.log.gz 2026-02-22T23:34:56.989 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32090.log 2026-02-22T23:34:56.989 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28299.log.gz 2026-02-22T23:34:56.989 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44153.log.gz 2026-02-22T23:34:56.989 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27515.log 2026-02-22T23:34:56.989 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29860.log 2026-02-22T23:34:56.989 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32090.log: /var/log/ceph/ceph-client.admin.27515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32090.log.gz 2026-02-22T23:34:56.989 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27515.log.gz 2026-02-22T23:34:56.989 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41248.log 2026-02-22T23:34:56.989 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27386.log 2026-02-22T23:34:56.989 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29860.log.gz 2026-02-22T23:34:56.989 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41248.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41248.log.gz 2026-02-22T23:34:56.989 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36466.log 2026-02-22T23:34:56.990 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25367.log 2026-02-22T23:34:56.990 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27386.log.gz 2026-02-22T23:34:56.990 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41247.log 2026-02-22T23:34:56.990 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36466.log.gz 2026-02-22T23:34:56.990 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.25367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43249.log 2026-02-22T23:34:56.990 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25367.log.gz 2026-02-22T23:34:56.990 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43821.log 2026-02-22T23:34:56.990 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41247.log.gz 2026-02-22T23:34:56.990 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41922.log 2026-02-22T23:34:56.990 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43249.log.gz 2026-02-22T23:34:56.990 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31648.log 2026-02-22T23:34:56.990 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43821.log.gz 2026-02-22T23:34:56.990 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42468.log 2026-02-22T23:34:56.990 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41922.log.gz 2026-02-22T23:34:56.991 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43353.log 2026-02-22T23:34:56.991 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31648.log.gz 2026-02-22T23:34:56.991 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35892.log 2026-02-22T23:34:56.991 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42468.log.gz 2026-02-22T23:34:56.991 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24645.log 2026-02-22T23:34:56.991 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43353.log.gz 2026-02-22T23:34:56.991 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30764.log 2026-02-22T23:34:56.991 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35892.log: /var/log/ceph/ceph-client.admin.24645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35892.log.gz 2026-02-22T23:34:56.991 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24645.log.gz 2026-02-22T23:34:56.991 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17291.log 2026-02-22T23:34:56.991 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30764.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.30764.log.gz -5 2026-02-22T23:34:56.991 INFO:teuthology.orchestra.run.trial115.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41479.log 2026-02-22T23:34:56.992 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.17291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42119.log 2026-02-22T23:34:56.992 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17291.log.gz 2026-02-22T23:34:56.992 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45595.log 2026-02-22T23:34:56.992 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41479.log: /var/log/ceph/ceph-client.admin.42119.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41479.log.gz 2026-02-22T23:34:56.992 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42119.log.gz 2026-02-22T23:34:56.992 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27957.log 2026-02-22T23:34:56.992 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29430.log 2026-02-22T23:34:56.992 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45595.log.gz 2026-02-22T23:34:56.992 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43211.log 2026-02-22T23:34:56.992 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27957.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27957.log.gz 2026-02-22T23:34:56.992 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29430.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29430.log.gz 2026-02-22T23:34:56.992 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32186.log 2026-02-22T23:34:57.014 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44935.log 2026-02-22T23:34:57.014 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43211.log.gz 2026-02-22T23:34:57.014 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32186.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32186.log.gz 2026-02-22T23:34:57.014 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40592.log 2026-02-22T23:34:57.014 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42469.log 2026-02-22T23:34:57.014 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44935.log.gz 2026-02-22T23:34:57.014 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.40592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32973.log 2026-02-22T23:34:57.014 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40592.log.gz 2026-02-22T23:34:57.014 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22280.log 2026-02-22T23:34:57.014 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42469.log.gz 2026-02-22T23:34:57.014 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29755.log 2026-02-22T23:34:57.014 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32973.log.gz 2026-02-22T23:34:57.014 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32058.log 2026-02-22T23:34:57.014 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22280.log.gz 2026-02-22T23:34:57.014 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29755.log.gz 2026-02-22T23:34:57.014 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28943.log 2026-02-22T23:34:57.014 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30739.log 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32058.log.gz 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41371.log 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28943.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28943.log.gz 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30739.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30739.log.gz 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33344.log 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41371.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.41371.log.gz -- 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr: /var/log/ceph/ceph-client.admin.28970.log 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41885.log 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33344.log.gz 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24674.log 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28970.log.gz 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44367.log 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41885.log.gz 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45752.log 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24674.log.gz 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36069.log 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44367.log.gz 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35339.log 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45752.log.gz 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41141.log 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36069.log.gz 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31418.log 2026-02-22T23:34:57.015 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35339.log: /var/log/ceph/ceph-client.admin.41141.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35339.log.gz 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41141.log.gz 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28064.log 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42152.log 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31418.log.gz 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr:gzip/var/log/ceph/ceph-client.admin.28064.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.24750.log 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28064.log.gz 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42152.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42152.log.gz 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21772.log 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32534.log 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24750.log.gz 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31824.log 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21772.log.gz 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32534.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32534.log.gz 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36095.log 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31966.log 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31824.log.gz 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17435.log 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36095.log.gz 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31966.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31966.log.gz 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44807.log 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31178.log 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.17435.log: /var/log/ceph/ceph-client.admin.44807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17435.log.gz 2026-02-22T23:34:57.016 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44807.log.gz 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21656.log 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31178.log.gz 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43445.log 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41244.log 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21656.log.gz 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28392.log 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43445.log.gz 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27490.log 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41244.log.gz 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28392.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28392.log.gz 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43818.log 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28403.log 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27490.log.gz 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43818.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43818.log.gz 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32738.log 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42123.log 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28403.log.gz 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32738.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32738.log.gz 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42223.log 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42277.log 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42123.log.gz 2026-02-22T23:34:57.017 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29520.log 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42223.log.gz 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30091.log 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42277.log.gz 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29520.log.gz 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43248.log 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36821.log 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30091.log: /var/log/ceph/ceph-client.admin.43248.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30091.log.gz 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43248.log.gz 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30398.log 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36821.log.gz 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32838.log 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29756.log 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30398.log.gz 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29618.log 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32838.log.gz 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29756.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29756.log.gz 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33239.log 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30976.log 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29618.log.gz 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33239.log.gz 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44909.log 2026-02-22T23:34:57.018 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30940.log/var/log/ceph/ceph-client.admin.30976.log: 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30976.log.gz 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22108.log 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44909.log.gz 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr:gzip/var/log/ceph/ceph-client.admin.30940.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33133.log 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30940.log.gz 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22108.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.22108.log.gz -5 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr: --verbose -- /var/log/ceph/ceph-client.admin.22231.log 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28845.log 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33133.log.gz 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22231.log.gz 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27623.log 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28845.log.gz 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21199.log 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44125.log 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27623.log.gz 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43003.log 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21199.log.gz 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44125.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.44125.log.gz /var/log/ceph/ceph-client.admin.27958.log 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr: 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43107.log 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43003.log.gz 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27958.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27958.log.gz 2026-02-22T23:34:57.019 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45804.log 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31859.log 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43107.log.gz 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42457.log 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45804.log.gz 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31859.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31859.log.gz 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30533.log 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42457.log.gz 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32942.log 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41339.log 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30533.log.gz 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24638.log 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32942.log.gz 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42911.log 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41339.log.gz 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24638.log.gz 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31306.log 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42911.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42911.log.gz 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35567.log 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42365.log 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31306.log.gz 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35567.log.gz 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46090.log 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25113.log 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42365.log: /var/log/ceph/ceph-client.admin.46090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42365.log.gz 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46090.log.gz 2026-02-22T23:34:57.020 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28508.log 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.25113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25138.log 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25113.log.gz 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28508.log.gz 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36405.log 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27726.log 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.25138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25138.log.gz 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28195.log 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36405.log.gz 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27726.log.gzgzip 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45543.log 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36183.log 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28195.log.gz 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr:gzip/var/log/ceph/ceph-client.admin.45543.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31523.log 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45543.log.gz 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31073.log 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36183.log.gz 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32533.log 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31523.log.gz 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44467.log 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31073.log.gz 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32533.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32533.log.gz 2026-02-22T23:34:57.021 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24544.log 2026-02-22T23:34:57.028 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44467.log.gz 2026-02-22T23:34:57.028 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43713.log 2026-02-22T23:34:57.028 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24544.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24544.log.gz 2026-02-22T23:34:57.028 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42029.log 2026-02-22T23:34:57.028 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31862.log 2026-02-22T23:34:57.028 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43713.log.gz 2026-02-22T23:34:57.028 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42029.log.gz 2026-02-22T23:34:57.028 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30978.log 2026-02-22T23:34:57.028 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31862.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31862.log.gz -5 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr: --verbose -- /var/log/ceph/ceph-client.admin.31081.log 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30978.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.30978.log.gz -5 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr: --verbose -- /var/log/ceph/ceph-client.admin.28834.log 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31081.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31081.log.gz 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29279.log 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28300.log 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28834.log.gz 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27221.log 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29279.log.gz 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28300.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28300.log.gz 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21878.log 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22179.log 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27221.log: /var/log/ceph/ceph-client.admin.21878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27221.log.gz 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21878.log.gz 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17679.log 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29854.log 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22179.log.gz 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.17679.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17679.log.gz 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29854.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29854.log.gz 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36567.log 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29952.log 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.36567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36567.log.gz 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41920.log 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29952.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29952.log.gz 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40597.log 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46116.log 2026-02-22T23:34:57.029 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41920.log.gz 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.40597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24846.log 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40597.log.gz 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.46116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46116.log.gz 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29519.log 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24846.log.gz 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29180.log 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43578.log 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29519.log: /var/log/ceph/ceph-client.admin.29180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29519.log.gz 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29180.log.gz 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22487.log 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43578.log.gz 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28600.log 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45569.log 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22487.log: /var/log/ceph/ceph-client.admin.28600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22487.log.gz 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28600.log.gz 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29484.log 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45569.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45569.log.gz 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28635.log 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31278.log 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29484.log: /var/log/ceph/ceph-client.admin.28635.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29484.log.gz 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28635.log.gz 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22257.log 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31278.log.gz 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44028.log 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24378.log 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22257.log.gz 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21200.log 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44028.log.gz 2026-02-22T23:34:57.030 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24378.log.gz 2026-02-22T23:34:57.031 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29078.log 2026-02-22T23:34:57.031 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30640.log 2026-02-22T23:34:57.031 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21200.log.gz 2026-02-22T23:34:57.031 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29962.log 2026-02-22T23:34:57.031 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29078.log.gz 2026-02-22T23:34:57.031 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30640.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30640.log.gz 2026-02-22T23:34:57.031 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32396.log 2026-02-22T23:34:57.031 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43139.log 2026-02-22T23:34:57.031 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29962.log.gz 2026-02-22T23:34:57.031 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32082.log 2026-02-22T23:34:57.031 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32396.log.gz 2026-02-22T23:34:57.031 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24775.log 2026-02-22T23:34:57.031 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43139.log.gz 2026-02-22T23:34:57.031 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40693.log 2026-02-22T23:34:57.031 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32082.log.gz 2026-02-22T23:34:57.031 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22360.log 2026-02-22T23:34:57.031 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24775.log.gz 2026-02-22T23:34:57.033 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.40693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40693.log.gz 2026-02-22T23:34:57.033 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30303.log 2026-02-22T23:34:57.033 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21552.log 2026-02-22T23:34:57.033 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22360.log: 0.0%/var/log/ceph/ceph-client.admin.30303.log: -- replaced with /var/log/ceph/ceph-client.admin.22360.log.gz 2026-02-22T23:34:57.033 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30303.log.gz 2026-02-22T23:34:57.033 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31620.log 2026-02-22T23:34:57.033 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21552.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21552.log.gz 2026-02-22T23:34:57.033 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21975.log 2026-02-22T23:34:57.033 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45308.log 2026-02-22T23:34:57.033 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31620.log.gz 2026-02-22T23:34:57.033 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21975.log.gz 2026-02-22T23:34:57.033 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40826.log 2026-02-22T23:34:57.033 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45308.log.gz 2026-02-22T23:34:57.033 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17602.log 2026-02-22T23:34:57.033 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28741.log 2026-02-22T23:34:57.034 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.40826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40826.log.gz 2026-02-22T23:34:57.034 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.17602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30197.log 2026-02-22T23:34:57.034 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17602.log.gz 2026-02-22T23:34:57.034 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28741.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28741.log.gz 2026-02-22T23:34:57.034 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28054.log 2026-02-22T23:34:57.034 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41235.log 2026-02-22T23:34:57.034 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30197.log.gz 2026-02-22T23:34:57.034 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28193.log 2026-02-22T23:34:57.034 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28054.log.gz 2026-02-22T23:34:57.034 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41235.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41235.log.gz 2026-02-22T23:34:57.034 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21407.log 2026-02-22T23:34:57.034 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22106.log 2026-02-22T23:34:57.034 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28193.log.gz 2026-02-22T23:34:57.035 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30292.log 2026-02-22T23:34:57.035 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21407.log.gz 2026-02-22T23:34:57.035 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22106.log.gz 2026-02-22T23:34:57.035 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21983.log 2026-02-22T23:34:57.035 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44121.log 2026-02-22T23:34:57.035 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30292.log.gz 2026-02-22T23:34:57.035 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21983.log.gz 2026-02-22T23:34:57.035 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35840.log 2026-02-22T23:34:57.035 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44121.log.gz 2026-02-22T23:34:57.035 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30975.log 2026-02-22T23:34:57.035 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44459.log 2026-02-22T23:34:57.035 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35840.log.gz 2026-02-22T23:34:57.036 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41791.log 2026-02-22T23:34:57.036 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30975.log.gz 2026-02-22T23:34:57.036 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44459.log.gz 2026-02-22T23:34:57.036 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29295.log 2026-02-22T23:34:57.036 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41791.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-22T23:34:57.037 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41791.log.gz 2026-02-22T23:34:57.037 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45856.log 2026-02-22T23:34:57.037 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29295.log.gz 2026-02-22T23:34:57.037 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43482.log 2026-02-22T23:34:57.037 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45856.log.gz 2026-02-22T23:34:57.038 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27725.log 2026-02-22T23:34:57.038 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43482.log.gz 2026-02-22T23:34:57.038 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41370.log 2026-02-22T23:34:57.038 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41788.log 2026-02-22T23:34:57.038 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27725.log.gz 2026-02-22T23:34:57.039 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41370.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.33462.log 2026-02-22T23:34:57.039 INFO:teuthology.orchestra.run.trial115.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41370.log.gz 2026-02-22T23:34:57.039 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41788.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41788.log.gz -5 2026-02-22T23:34:57.039 INFO:teuthology.orchestra.run.trial115.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44667.log 2026-02-22T23:34:57.040 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33462.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.33462.log.gz --verbose 2026-02-22T23:34:57.040 INFO:teuthology.orchestra.run.trial115.stderr: -- /var/log/ceph/ceph-client.admin.44262.log 2026-02-22T23:34:57.040 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42015.log 2026-02-22T23:34:57.040 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44667.log.gz 2026-02-22T23:34:57.040 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44262.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44262.log.gz 2026-02-22T23:34:57.040 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44154.log 2026-02-22T23:34:57.041 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40559.log 2026-02-22T23:34:57.041 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42015.log.gz 2026-02-22T23:34:57.041 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21198.log 2026-02-22T23:34:57.041 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44154.log.gz 2026-02-22T23:34:57.042 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.40559.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.40559.log.gz -- 2026-02-22T23:34:57.042 INFO:teuthology.orchestra.run.trial115.stderr: /var/log/ceph/ceph-client.admin.36568.log 2026-02-22T23:34:57.042 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27622.log 2026-02-22T23:34:57.042 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21198.log.gz 2026-02-22T23:34:57.042 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36568.log.gz 2026-02-22T23:34:57.043 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42565.log 2026-02-22T23:34:57.043 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32405.log 2026-02-22T23:34:57.043 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27622.log.gz 2026-02-22T23:34:57.043 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42565.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42565.log.gz 2026-02-22T23:34:57.043 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43347.log 2026-02-22T23:34:57.043 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45008.log 2026-02-22T23:34:57.044 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32405.log: /var/log/ceph/ceph-client.admin.43347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32405.log.gz 2026-02-22T23:34:57.044 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43347.log.gz 2026-02-22T23:34:57.044 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31283.log 2026-02-22T23:34:57.044 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41688.log 2026-02-22T23:34:57.044 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45008.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45008.log.gz 2026-02-22T23:34:57.044 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31283.log.gz 2026-02-22T23:34:57.045 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30844.log 2026-02-22T23:34:57.045 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41688.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41688.log.gz 2026-02-22T23:34:57.045 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43589.log 2026-02-22T23:34:57.045 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28637.log 2026-02-22T23:34:57.045 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30844.log.gz 2026-02-22T23:34:57.046 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35738.log 2026-02-22T23:34:57.046 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43589.log.gz 2026-02-22T23:34:57.046 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41134.log 2026-02-22T23:34:57.046 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28637.log.gz 2026-02-22T23:34:57.046 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28506.log 2026-02-22T23:34:57.046 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35738.log.gz 2026-02-22T23:34:57.047 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21526.log 2026-02-22T23:34:57.047 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41134.log.gz 2026-02-22T23:34:57.047 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28506.log.gz 2026-02-22T23:34:57.047 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27388.log 2026-02-22T23:34:57.047 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25114.log 2026-02-22T23:34:57.048 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21526.log.gz 2026-02-22T23:34:57.048 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36795.log 2026-02-22T23:34:57.048 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27388.log.gz 2026-02-22T23:34:57.048 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.25114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43819.log 2026-02-22T23:34:57.048 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25114.log.gz 2026-02-22T23:34:57.049 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21974.log 2026-02-22T23:34:57.049 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36795.log.gz 2026-02-22T23:34:57.049 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31417.log 2026-02-22T23:34:57.049 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43819.log.gz 2026-02-22T23:34:57.049 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22561.log 2026-02-22T23:34:57.049 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21974.log.gz 2026-02-22T23:34:57.050 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31417.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31417.log.gz 2026-02-22T23:34:57.050 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36181.log 2026-02-22T23:34:57.050 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22561.log.gz 2026-02-22T23:34:57.050 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22205.log 2026-02-22T23:34:57.050 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36181.log.gz 2026-02-22T23:34:57.050 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42327.log 2026-02-22T23:34:57.051 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27247.log 2026-02-22T23:34:57.051 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22205.log.gz 2026-02-22T23:34:57.051 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42470.log 2026-02-22T23:34:57.051 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42327.log.gz 2026-02-22T23:34:57.051 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27247.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27247.log.gz 2026-02-22T23:34:57.051 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27283.log 2026-02-22T23:34:57.051 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35609.log 2026-02-22T23:34:57.052 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42470.log.gz 2026-02-22T23:34:57.052 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27283.log.gz 2026-02-22T23:34:57.052 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25314.log 2026-02-22T23:34:57.052 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21500.log 2026-02-22T23:34:57.052 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35609.log.gz 2026-02-22T23:34:57.053 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44261.log 2026-02-22T23:34:57.053 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.25314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25314.log.gz 2026-02-22T23:34:57.053 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25393.log 2026-02-22T23:34:57.053 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21500.log.gz 2026-02-22T23:34:57.053 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44261.log.gz 2026-02-22T23:34:57.053 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28295.log 2026-02-22T23:34:57.053 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29183.log 2026-02-22T23:34:57.053 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.25393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25393.log.gz 2026-02-22T23:34:57.054 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28295.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28295.log.gz 2026-02-22T23:34:57.054 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36284.log 2026-02-22T23:34:57.054 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29280.log 2026-02-22T23:34:57.054 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29183.log.gz 2026-02-22T23:34:57.055 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30198.log 2026-02-22T23:34:57.055 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36284.log.gz 2026-02-22T23:34:57.055 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24543.log 2026-02-22T23:34:57.055 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29280.log.gz 2026-02-22T23:34:57.055 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30198.log.gz 2026-02-22T23:34:57.055 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41689.log 2026-02-22T23:34:57.055 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42807.log 2026-02-22T23:34:57.056 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24543.log.gz 2026-02-22T23:34:57.056 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44936.log 2026-02-22T23:34:57.056 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41689.log.gz 2026-02-22T23:34:57.056 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45438.log 2026-02-22T23:34:57.056 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42807.log.gz 2026-02-22T23:34:57.056 INFO:teuthology.orchestra.run.trial115.stderr:gzip/var/log/ceph/ceph-client.admin.44936.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31226.log 2026-02-22T23:34:57.057 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44936.log.gz 2026-02-22T23:34:57.057 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45438.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.45438.log.gz -- 2026-02-22T23:34:57.057 INFO:teuthology.orchestra.run.trial115.stderr: /var/log/ceph/ceph-client.admin.41039.log 2026-02-22T23:34:57.057 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31226.log.gz 2026-02-22T23:34:57.057 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24909.log 2026-02-22T23:34:57.057 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41039.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41039.log.gz 2026-02-22T23:34:57.057 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44471.log 2026-02-22T23:34:57.058 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43247.log 2026-02-22T23:34:57.058 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24909.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24909.log.gz 2026-02-22T23:34:57.058 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42153.log 2026-02-22T23:34:57.058 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44471.log.gz 2026-02-22T23:34:57.058 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43247.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43247.log.gz 2026-02-22T23:34:57.058 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41921.log 2026-02-22T23:34:57.059 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30498.log 2026-02-22T23:34:57.059 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42153.log.gz 2026-02-22T23:34:57.059 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41921.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41921.log.gz 2026-02-22T23:34:57.059 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45360.log 2026-02-22T23:34:57.059 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45674.log 2026-02-22T23:34:57.060 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30498.log.gz 2026-02-22T23:34:57.060 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45360.log.gz 2026-02-22T23:34:57.060 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29522.log 2026-02-22T23:34:57.060 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31861.log 2026-02-22T23:34:57.060 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45674.log.gz 2026-02-22T23:34:57.060 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44029.log 2026-02-22T23:34:57.061 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29522.log.gz 2026-02-22T23:34:57.061 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29308.log 2026-02-22T23:34:57.061 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31861.log.gz 2026-02-22T23:34:57.061 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44029.log.gz 2026-02-22T23:34:57.061 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28158.log 2026-02-22T23:34:57.061 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29308.log.gz 2026-02-22T23:34:57.061 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29958.log 2026-02-22T23:34:57.062 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32089.log 2026-02-22T23:34:57.062 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28158.log.gz 2026-02-22T23:34:57.062 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44364.log 2026-02-22T23:34:57.062 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29958.log.gz 2026-02-22T23:34:57.063 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38690.log 2026-02-22T23:34:57.063 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32089.log.gz 2026-02-22T23:34:57.063 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44364.log.gz 2026-02-22T23:34:57.063 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32303.log 2026-02-22T23:34:57.063 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.38690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29384.log 2026-02-22T23:34:57.064 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38690.log.gz 2026-02-22T23:34:57.064 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31382.log 2026-02-22T23:34:57.064 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32303.log.gz 2026-02-22T23:34:57.064 INFO:teuthology.orchestra.run.trial115.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2026-02-22T23:34:57.064 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43579.log 2026-02-22T23:34:57.064 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29384.log.gz 2026-02-22T23:34:57.064 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33136.log 2026-02-22T23:34:57.065 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31382.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31382.log.gz 2026-02-22T23:34:57.065 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43579.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43579.log.gz 2026-02-22T23:34:57.065 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36573.log 2026-02-22T23:34:57.065 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33136.log.gz 2026-02-22T23:34:57.065 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24907.log 2026-02-22T23:34:57.065 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36573.log.gz 2026-02-22T23:34:57.065 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27857.log 2026-02-22T23:34:57.066 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24907.log.gz 2026-02-22T23:34:57.066 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27391.log 2026-02-22T23:34:57.066 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27857.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27857.log.gz 2026-02-22T23:34:57.066 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36668.log 2026-02-22T23:34:57.066 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27391.log.gz 2026-02-22T23:34:57.066 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32191.log 2026-02-22T23:34:57.066 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36668.log.gz 2026-02-22T23:34:57.066 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45934.log 2026-02-22T23:34:57.067 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32191.log.gz 2026-02-22T23:34:57.067 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35969.log 2026-02-22T23:34:57.067 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36569.log 2026-02-22T23:34:57.067 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45934.log.gz 2026-02-22T23:34:57.067 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17491.log 2026-02-22T23:34:57.068 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35969.log.gz 2026-02-22T23:34:57.068 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42693.log 2026-02-22T23:34:57.068 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36569.log.gz 2026-02-22T23:34:57.068 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.17491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40596.log 2026-02-22T23:34:57.068 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17491.log.gz 2026-02-22T23:34:57.069 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29625.log 2026-02-22T23:34:57.069 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42693.log.gz 2026-02-22T23:34:57.069 INFO:teuthology.orchestra.run.trial115.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-22T23:34:57.069 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.40596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27482.log 2026-02-22T23:34:57.069 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40596.log.gz 2026-02-22T23:34:57.069 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30628.log 2026-02-22T23:34:57.069 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29625.log.gz 2026-02-22T23:34:57.070 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21774.log 2026-02-22T23:34:57.070 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27482.log.gz 2026-02-22T23:34:57.070 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28507.log 2026-02-22T23:34:57.070 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30628.log.gz 2026-02-22T23:34:57.070 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33343.log 2026-02-22T23:34:57.071 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21774.log.gz 2026-02-22T23:34:57.071 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30394.log 2026-02-22T23:34:57.071 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28507.log.gz 2026-02-22T23:34:57.071 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33436.log 2026-02-22T23:34:57.071 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33343.log.gz 2026-02-22T23:34:57.072 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30394.log.gz 2026-02-22T23:34:57.072 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33342.log 2026-02-22T23:34:57.072 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33436.log.gz 2026-02-22T23:34:57.072 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22461.log 2026-02-22T23:34:57.072 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33342.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.33342.log.gz 2026-02-22T23:34:57.072 INFO:teuthology.orchestra.run.trial115.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30056.log 2026-02-22T23:34:57.072 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22461.log.gz 2026-02-22T23:34:57.073 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40897.log 2026-02-22T23:34:57.073 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35741.log 2026-02-22T23:34:57.073 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30056.log.gz 2026-02-22T23:34:57.073 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.40897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40897.log.gz 2026-02-22T23:34:57.073 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41142.log 2026-02-22T23:34:57.074 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44700.log 2026-02-22T23:34:57.074 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35741.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35741.log.gz 2026-02-22T23:34:57.074 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35945.log 2026-02-22T23:34:57.074 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41142.log.gz 2026-02-22T23:34:57.074 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44700.log.gz 2026-02-22T23:34:57.074 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46064.log 2026-02-22T23:34:57.075 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30428.log 2026-02-22T23:34:57.075 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35945.log.gz 2026-02-22T23:34:57.075 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.46064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31963.log 0.0% 2026-02-22T23:34:57.075 INFO:teuthology.orchestra.run.trial115.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46064.log.gz 2026-02-22T23:34:57.075 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30428.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.41577.log -- replaced with /var/log/ceph/ceph-client.admin.30428.log.gz 2026-02-22T23:34:57.075 INFO:teuthology.orchestra.run.trial115.stderr: 2026-02-22T23:34:57.076 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29754.log 2026-02-22T23:34:57.076 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31963.log.gz 2026-02-22T23:34:57.076 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41143.log 2026-02-22T23:34:57.076 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41577.log.gz 2026-02-22T23:34:57.076 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29754.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29754.log.gz 2026-02-22T23:34:57.076 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45465.log 2026-02-22T23:34:57.077 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35403.log 2026-02-22T23:34:57.077 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41143.log.gz 2026-02-22T23:34:57.077 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45465.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45465.log.gz 2026-02-22T23:34:57.077 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44701.log 2026-02-22T23:34:57.078 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28196.log 2026-02-22T23:34:57.078 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35403.log.gz 2026-02-22T23:34:57.078 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30092.log 2026-02-22T23:34:57.078 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44701.log.gz 2026-02-22T23:34:57.078 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44805.log 2026-02-22T23:34:57.078 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28196.log.gz 2026-02-22T23:34:57.079 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30092.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30092.log.gzgzip 2026-02-22T23:34:57.079 INFO:teuthology.orchestra.run.trial115.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.27285.log 2026-02-22T23:34:57.079 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44805.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44805.log.gz 2026-02-22T23:34:57.079 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46038.log 2026-02-22T23:34:57.079 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27285.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27285.log.gz 2026-02-22T23:34:57.080 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32639.log 2026-02-22T23:34:57.080 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.46038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46038.log.gz 2026-02-22T23:34:57.080 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35832.log 2026-02-22T23:34:57.080 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32639.log.gz 2026-02-22T23:34:57.080 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24439.log 2026-02-22T23:34:57.081 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29521.log 2026-02-22T23:34:57.081 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35832.log.gz 2026-02-22T23:34:57.081 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28636.log 2026-02-22T23:34:57.081 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24439.log.gz 2026-02-22T23:34:57.081 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31616.log 2026-02-22T23:34:57.081 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29521.log.gz 2026-02-22T23:34:57.082 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28636.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28636.log.gz 2026-02-22T23:34:57.082 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21604.log 2026-02-22T23:34:57.082 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27489.log 2026-02-22T23:34:57.082 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31616.log.gz 2026-02-22T23:34:57.082 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43924.log 2026-02-22T23:34:57.083 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21604.log.gz 2026-02-22T23:34:57.083 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28969.log 2026-02-22T23:34:57.083 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27489.log.gz 2026-02-22T23:34:57.083 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43924.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43924.log.gz 2026-02-22T23:34:57.083 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30845.log 2026-02-22T23:34:57.084 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42804.log 2026-02-22T23:34:57.084 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28969.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28969.log.gz 2026-02-22T23:34:57.084 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30845.log.gz 2026-02-22T23:34:57.084 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25014.log 2026-02-22T23:34:57.084 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36769.log 2026-02-22T23:34:57.084 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42804.log.gz 2026-02-22T23:34:57.085 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46167.log 2026-02-22T23:34:57.085 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.25014.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25014.log.gz 2026-02-22T23:34:57.085 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32409.log 2026-02-22T23:34:57.085 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36769.log.gz 2026-02-22T23:34:57.085 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.46167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46167.log.gz 2026-02-22T23:34:57.085 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41036.log 2026-02-22T23:34:57.086 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43035.log 2026-02-22T23:34:57.086 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32409.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32409.log.gz 2026-02-22T23:34:57.086 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41036.log.gz 2026-02-22T23:34:57.086 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36041.log 2026-02-22T23:34:57.086 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22075.log 2026-02-22T23:34:57.087 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43035.log.gz 2026-02-22T23:34:57.087 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36041.log.gz 2026-02-22T23:34:57.087 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35737.log 2026-02-22T23:34:57.087 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22075.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22075.log.gz 2026-02-22T23:34:57.087 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43131.log 2026-02-22T23:34:57.088 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32974.log 2026-02-22T23:34:57.088 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35737.log.gz 2026-02-22T23:34:57.088 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36431.log 2026-02-22T23:34:57.088 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43131.log.gz 2026-02-22T23:34:57.088 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32974.log.gz 2026-02-22T23:34:57.088 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43922.log 2026-02-22T23:34:57.089 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36431.log.gz 2026-02-22T23:34:57.089 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22079.log 2026-02-22T23:34:57.089 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35402.log 2026-02-22T23:34:57.089 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43922.log.gz 2026-02-22T23:34:57.089 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17539.log 2026-02-22T23:34:57.090 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22079.log: /var/log/ceph/ceph-client.admin.35402.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22079.log.gz 2026-02-22T23:34:57.090 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35402.log.gz 2026-02-22T23:34:57.090 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30302.log 2026-02-22T23:34:57.090 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.17539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42027.log 2026-02-22T23:34:57.090 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17539.log.gz 2026-02-22T23:34:57.090 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30302.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30302.log.gz 2026-02-22T23:34:57.090 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24440.log 2026-02-22T23:34:57.091 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46012.log 2026-02-22T23:34:57.091 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42027.log.gz 2026-02-22T23:34:57.091 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32750.log 2026-02-22T23:34:57.091 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24440.log.gz 2026-02-22T23:34:57.091 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.46012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28194.log 2026-02-22T23:34:57.091 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46012.log.gz 2026-02-22T23:34:57.092 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32750.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32750.log.gz 2026-02-22T23:34:57.092 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40700.log 2026-02-22T23:34:57.092 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31751.log 2026-02-22T23:34:57.092 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28194.log.gz 2026-02-22T23:34:57.092 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.40700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40700.log.gz 2026-02-22T23:34:57.092 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21300.log 2026-02-22T23:34:57.093 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30536.log 2026-02-22T23:34:57.093 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31751.log.gz 2026-02-22T23:34:57.093 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21300.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-22T23:34:57.093 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21300.log.gz 2026-02-22T23:34:57.094 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24872.log 2026-02-22T23:34:57.094 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30536.log.gz 2026-02-22T23:34:57.094 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28844.log 2026-02-22T23:34:57.094 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24872.log.gz 2026-02-22T23:34:57.094 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27624.log 2026-02-22T23:34:57.095 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28844.log.gz 2026-02-22T23:34:57.095 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42669.log 2026-02-22T23:34:57.095 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33098.log 2026-02-22T23:34:57.095 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27624.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27624.log.gz 2026-02-22T23:34:57.095 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42669.log.gz 2026-02-22T23:34:57.095 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32641.log 2026-02-22T23:34:57.096 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31516.log 2026-02-22T23:34:57.096 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33098.log.gz 2026-02-22T23:34:57.096 INFO:teuthology.orchestra.run.trial115.stderr: 95.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-22T23:34:57.096 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31074.log 2026-02-22T23:34:57.096 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32641.log.gz 2026-02-22T23:34:57.096 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32946.log 2026-02-22T23:34:57.096 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31516.log.gz 2026-02-22T23:34:57.096 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31074.log.gz 2026-02-22T23:34:57.097 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30740.log 2026-02-22T23:34:57.097 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32946.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32946.log.gz 2026-02-22T23:34:57.097 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44705.log 2026-02-22T23:34:57.097 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33238.log 2026-02-22T23:34:57.097 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30740.log.gz 2026-02-22T23:34:57.098 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44705.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-22T23:34:57.098 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44705.log.gz 2026-02-22T23:34:57.098 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21711.log 2026-02-22T23:34:57.098 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33238.log.gz 2026-02-22T23:34:57.099 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30639.log 2026-02-22T23:34:57.099 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21711.log.gz 2026-02-22T23:34:57.099 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30294.log 2026-02-22T23:34:57.099 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30639.log.gz 2026-02-22T23:34:57.099 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21775.log 2026-02-22T23:34:57.099 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30294.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30294.log.gz 2026-02-22T23:34:57.100 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22535.log 2026-02-22T23:34:57.100 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41781.log 2026-02-22T23:34:57.100 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21775.log.gz 2026-02-22T23:34:57.100 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22535.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22535.log.gz 2026-02-22T23:34:57.100 INFO:teuthology.orchestra.run.trial115.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.21737.log 2026-02-22T23:34:57.101 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41781.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2026-02-22T23:34:57.101 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41781.log.gz 2026-02-22T23:34:57.101 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5/var/log/ceph/ceph-client.admin.21737.log: --verbose -- /var/log/ceph/ceph-client.admin.31647.log 2026-02-22T23:34:57.101 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21737.log.gz 2026-02-22T23:34:57.101 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36275.log 2026-02-22T23:34:57.102 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31647.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31647.log.gz 2026-02-22T23:34:57.102 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33072.log 2026-02-22T23:34:57.102 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40801.log 2026-02-22T23:34:57.102 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36275.log.gz 2026-02-22T23:34:57.102 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33072.log.gz 2026-02-22T23:34:57.103 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-22T23:34:57.103 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.40801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42665.log 2026-02-22T23:34:57.103 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40801.log.gz 2026-02-22T23:34:57.103 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30190.log 2026-02-22T23:34:57.104 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42594.log 2026-02-22T23:34:57.104 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42665.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42665.log.gz 2026-02-22T23:34:57.104 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30190.log.gz 2026-02-22T23:34:57.104 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31752.log 2026-02-22T23:34:57.104 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45986.log 2026-02-22T23:34:57.105 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42594.log.gz 2026-02-22T23:34:57.105 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31752.log.gz 2026-02-22T23:34:57.105 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35944.log 2026-02-22T23:34:57.105 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25006.log 2026-02-22T23:34:57.105 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45986.log.gz 2026-02-22T23:34:57.106 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22413.log 2026-02-22T23:34:57.106 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35944.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35944.log.gz 2026-02-22T23:34:57.106 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.25006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25006.log.gz 2026-02-22T23:34:57.106 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30836.log 2026-02-22T23:34:57.106 INFO:teuthology.orchestra.run.trial115.stderr: 94.3% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-22T23:34:57.106 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25005.log 2026-02-22T23:34:57.106 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22413.log.gz 2026-02-22T23:34:57.106 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30836.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30836.log.gz 2026-02-22T23:34:57.107 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21197.log 2026-02-22T23:34:57.107 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.25005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31419.log 2026-02-22T23:34:57.107 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25005.log.gz 2026-02-22T23:34:57.107 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27856.log 2026-02-22T23:34:57.107 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21197.log.gz 2026-02-22T23:34:57.108 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29080.log 2026-02-22T23:34:57.108 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31419.log.gz 2026-02-22T23:34:57.108 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27856.log.gz 2026-02-22T23:34:57.108 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28846.log 2026-02-22T23:34:57.108 INFO:teuthology.orchestra.run.trial115.stderr:gzip/var/log/ceph/ceph-client.admin.29080.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42717.log 2026-02-22T23:34:57.108 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29080.log.gz 2026-02-22T23:34:57.109 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28846.log.gz 2026-02-22T23:34:57.109 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30535.log 2026-02-22T23:34:57.109 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36180.log 2026-02-22T23:34:57.109 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42717.log.gz 2026-02-22T23:34:57.109 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30535.log.gz 2026-02-22T23:34:57.109 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41037.log 2026-02-22T23:34:57.110 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29856.log 2026-02-22T23:34:57.110 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36180.log.gz 2026-02-22T23:34:57.110 INFO:teuthology.orchestra.run.trial115.stderr:gzip/var/log/ceph/ceph-client.admin.41037.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40929.log 2026-02-22T23:34:57.110 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41037.log.gz 2026-02-22T23:34:57.110 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29856.log.gz 2026-02-22T23:34:57.110 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33228.log 2026-02-22T23:34:57.111 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32304.log 2026-02-22T23:34:57.111 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.40929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40929.log.gz 2026-02-22T23:34:57.111 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33228.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33228.log.gz 2026-02-22T23:34:57.111 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41574.log 2026-02-22T23:34:57.111 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32304.log.gz 2026-02-22T23:34:57.111 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21630.log 2026-02-22T23:34:57.111 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41574.log.gz 2026-02-22T23:34:57.112 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28065.log 2026-02-22T23:34:57.112 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31420.log 2026-02-22T23:34:57.112 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21630.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21630.log.gz 2026-02-22T23:34:57.112 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28065.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28065.log.gz 2026-02-22T23:34:57.112 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35631.log 2026-02-22T23:34:57.112 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31420.log.gz 2026-02-22T23:34:57.113 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27378.log 2026-02-22T23:34:57.113 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35631.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35631.log.gz 2026-02-22T23:34:57.113 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25210.log 2026-02-22T23:34:57.113 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27378.log.gz 2026-02-22T23:34:57.113 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30732.log 2026-02-22T23:34:57.113 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.25210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45034.log 2026-02-22T23:34:57.114 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25210.log.gz 2026-02-22T23:34:57.114 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25217.log 2026-02-22T23:34:57.114 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30732.log.gz 2026-02-22T23:34:57.114 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35841.log 2026-02-22T23:34:57.114 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45034.log.gz 2026-02-22T23:34:57.115 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.25217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32749.log 2026-02-22T23:34:57.115 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25217.log.gz 2026-02-22T23:34:57.115 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.35841.log: /var/log/ceph/ceph-client.admin.38716.log 2026-02-22T23:34:57.115 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35841.log.gz 2026-02-22T23:34:57.115 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32749.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32749.log.gz 2026-02-22T23:34:57.115 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41923.log 2026-02-22T23:34:57.116 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.38716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38716.log.gz 2026-02-22T23:34:57.116 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42806.log 2026-02-22T23:34:57.116 INFO:teuthology.orchestra.run.trial115.stderr:gzip/var/log/ceph/ceph-client.admin.41923.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36182.log 2026-02-22T23:34:57.116 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41923.log.gz 2026-02-22T23:34:57.116 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42806.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42806.log.gz 2026-02-22T23:34:57.116 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41001.log 2026-02-22T23:34:57.117 INFO:teuthology.orchestra.run.trial115.stderr:gzip/var/log/ceph/ceph-client.admin.36182.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41443.log 2026-02-22T23:34:57.117 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36182.log.gz 2026-02-22T23:34:57.117 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30189.log 2026-02-22T23:34:57.117 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41001.log.gz 2026-02-22T23:34:57.117 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41443.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41443.log.gz 2026-02-22T23:34:57.117 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21406.log 2026-02-22T23:34:57.117 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30189.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30189.log.gz 2026-02-22T23:34:57.118 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44641.log 2026-02-22T23:34:57.118 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31330.log 2026-02-22T23:34:57.118 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21406.log.gz 2026-02-22T23:34:57.118 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29718.log 2026-02-22T23:34:57.118 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44641.log.gz 2026-02-22T23:34:57.119 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31330.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31330.log.gz 2026-02-22T23:34:57.119 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25218.log 2026-02-22T23:34:57.119 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43012.log 2026-02-22T23:34:57.119 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29718.log.gz 2026-02-22T23:34:57.119 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.25218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42910.log 2026-02-22T23:34:57.120 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25218.log.gz 2026-02-22T23:34:57.120 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21302.log 2026-02-22T23:34:57.120 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43012.log.gz 2026-02-22T23:34:57.120 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42910.log.gz 2026-02-22T23:34:57.120 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42769.log 2026-02-22T23:34:57.121 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35400.log 2026-02-22T23:34:57.121 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21302.log.gz 2026-02-22T23:34:57.121 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28066.log 2026-02-22T23:34:57.121 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42769.log.gz 2026-02-22T23:34:57.121 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21396.log 2026-02-22T23:34:57.121 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35400.log.gz 2026-02-22T23:34:57.121 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35937.log 2026-02-22T23:34:57.122 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28066.log.gz 2026-02-22T23:34:57.122 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21396.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21396.log.gz 2026-02-22T23:34:57.122 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28402.log 2026-02-22T23:34:57.122 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44355.log 2026-02-22T23:34:57.122 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35937.log.gz 2026-02-22T23:34:57.122 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28402.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28402.log.gz 2026-02-22T23:34:57.122 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29383.log 2026-02-22T23:34:57.123 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43348.log 2026-02-22T23:34:57.123 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44355.log.gz 2026-02-22T23:34:57.123 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24910.log 2026-02-22T23:34:57.123 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29383.log.gz 2026-02-22T23:34:57.123 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25337.log 2026-02-22T23:34:57.123 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43348.log.gz 2026-02-22T23:34:57.124 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44615.log 2026-02-22T23:34:57.124 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24910.log.gz 2026-02-22T23:34:57.124 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.25337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29182.log 2026-02-22T23:34:57.124 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25337.log.gz 2026-02-22T23:34:57.124 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44468.log 2026-02-22T23:34:57.124 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44615.log.gz 2026-02-22T23:34:57.125 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36467.log 2026-02-22T23:34:57.125 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29182.log.gz 2026-02-22T23:34:57.125 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.44468.log: /var/log/ceph/ceph-client.admin.44589.log 2026-02-22T23:34:57.125 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44468.log.gz 2026-02-22T23:34:57.125 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36467.log.gz 2026-02-22T23:34:57.125 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30094.log 2026-02-22T23:34:57.126 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43138.log 2026-02-22T23:34:57.126 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44589.log.gz 2026-02-22T23:34:57.126 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30094.log.gz 2026-02-22T23:34:57.126 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28638.log 2026-02-22T23:34:57.126 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27286.log 2026-02-22T23:34:57.127 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43138.log.gz 2026-02-22T23:34:57.127 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28638.log.gz 2026-02-22T23:34:57.127 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42900.log 2026-02-22T23:34:57.127 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27286.log.gz 2026-02-22T23:34:57.127 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45908.log 2026-02-22T23:34:57.127 INFO:teuthology.orchestra.run.trial115.stderr:gzip/var/log/ceph/ceph-client.admin.42900.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31177.log 2026-02-22T23:34:57.128 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42900.log.gz 2026-02-22T23:34:57.128 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45908.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45908.log.gz 2026-02-22T23:34:57.128 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28404.log 2026-02-22T23:34:57.128 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.31177.log: /var/log/ceph/ceph-client.admin.27962.log 2026-02-22T23:34:57.128 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31177.log.gz 2026-02-22T23:34:57.128 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28404.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28404.log.gz 2026-02-22T23:34:57.128 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40928.log 2026-02-22T23:34:57.129 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40701.log 2026-02-22T23:34:57.129 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27962.log.gz 2026-02-22T23:34:57.129 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.40928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40928.log.gz 2026-02-22T23:34:57.129 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29420.log 2026-02-22T23:34:57.129 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.40701.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40701.log.gz 2026-02-22T23:34:57.129 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44704.log 2026-02-22T23:34:57.130 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29420.log.gz 2026-02-22T23:34:57.130 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21578.log 2026-02-22T23:34:57.130 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31524.log 2026-02-22T23:34:57.130 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44704.log.gz 2026-02-22T23:34:57.130 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21578.log.gz 2026-02-22T23:34:57.130 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45778.log 2026-02-22T23:34:57.131 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32642.log 0.0% 2026-02-22T23:34:57.131 INFO:teuthology.orchestra.run.trial115.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31524.log.gz 2026-02-22T23:34:57.131 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28740.log 2026-02-22T23:34:57.131 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45778.log.gz 2026-02-22T23:34:57.132 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45386.log 2026-02-22T23:34:57.132 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32642.log.gz 2026-02-22T23:34:57.132 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28740.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28740.log.gz 2026-02-22T23:34:57.132 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41677.log 2026-02-22T23:34:57.132 INFO:teuthology.orchestra.run.trial115.stderr:gzip/var/log/ceph/ceph-client.admin.45386.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41686.log 2026-02-22T23:34:57.132 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45386.log.gz 2026-02-22T23:34:57.133 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41677.log.gz 2026-02-22T23:34:57.133 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36287.log 2026-02-22T23:34:57.133 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41686.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41686.log.gz 2026-02-22T23:34:57.133 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36286.log 2026-02-22T23:34:57.133 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22309.log 2026-02-22T23:34:57.133 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36287.log.gz 2026-02-22T23:34:57.133 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36286.log.gz 2026-02-22T23:34:57.134 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43925.log 2026-02-22T23:34:57.134 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22309.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22309.log.gz 2026-02-22T23:34:57.134 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25013.log 2026-02-22T23:34:57.135 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44031.log 2026-02-22T23:34:57.135 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43925.log.gz 2026-02-22T23:34:57.135 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.25013.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25013.log.gz 2026-02-22T23:34:57.135 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33488.log 2026-02-22T23:34:57.135 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28739.log 2026-02-22T23:34:57.135 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44031.log.gz 2026-02-22T23:34:57.136 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41590.log 2026-02-22T23:34:57.136 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33488.log.gz 2026-02-22T23:34:57.136 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35739.log 2026-02-22T23:34:57.136 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28739.log.gz 2026-02-22T23:34:57.136 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41590.log.gz 2026-02-22T23:34:57.136 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42802.log 2026-02-22T23:34:57.137 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35739.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35739.log.gz 2026-02-22T23:34:57.137 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44225.log 2026-02-22T23:34:57.137 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42802.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42802.log.gz 2026-02-22T23:34:57.137 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42364.log 2026-02-22T23:34:57.138 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41478.log 2026-02-22T23:34:57.138 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44225.log.gz 2026-02-22T23:34:57.138 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28742.log 2026-02-22T23:34:57.138 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42364.log.gz 2026-02-22T23:34:57.138 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44260.log 2026-02-22T23:34:57.138 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41478.log.gz 2026-02-22T23:34:57.139 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28742.log.gz 2026-02-22T23:34:57.139 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27858.log 2026-02-22T23:34:57.139 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21982.log 2026-02-22T23:34:57.139 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44260.log.gz 2026-02-22T23:34:57.139 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27858.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.27858.log.gz /var/log/ceph/ceph-client.admin.35401.log 2026-02-22T23:34:57.140 INFO:teuthology.orchestra.run.trial115.stderr: 2026-02-22T23:34:57.140 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35365.log 2026-02-22T23:34:57.140 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21982.log.gz 2026-02-22T23:34:57.140 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35401.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.35401.log.gz 2026-02-22T23:34:57.140 INFO:teuthology.orchestra.run.trial115.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30977.log 2026-02-22T23:34:57.141 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36045.log 2026-02-22T23:34:57.141 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35365.log.gz 2026-02-22T23:34:57.141 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30977.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30977.log.gz 2026-02-22T23:34:57.141 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45517.log 2026-02-22T23:34:57.141 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36045.log.gz 2026-02-22T23:34:57.141 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44263.log 2026-02-22T23:34:57.141 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45517.log.gz 2026-02-22T23:34:57.142 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21304.log 2026-02-22T23:34:57.142 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44263.log.gz 2026-02-22T23:34:57.142 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44368.log 2026-02-22T23:34:57.142 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33135.log 2026-02-22T23:34:57.142 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21304.log.gz 2026-02-22T23:34:57.142 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44368.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44368.log.gz 2026-02-22T23:34:57.142 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43483.log 2026-02-22T23:34:57.143 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33135.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33135.log.gz 2026-02-22T23:34:57.143 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27820.log 2026-02-22T23:34:57.143 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40692.log/var/log/ceph/ceph-client.admin.43483.log: 2026-02-22T23:34:57.143 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43483.log.gz 2026-02-22T23:34:57.143 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27820.log.gz 2026-02-22T23:34:57.143 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-22T23:34:57.144 INFO:teuthology.orchestra.run.trial115.stderr:gzip/var/log/ceph/ceph-client.admin.40692.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41589.log 2026-02-22T23:34:57.144 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40692.log.gz 2026-02-22T23:34:57.144 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42026.log 2026-02-22T23:34:57.145 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41589.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41589.log.gz -5 2026-02-22T23:34:57.145 INFO:teuthology.orchestra.run.trial115.stderr: --verbose -- /var/log/ceph/ceph-client.admin.30093.log 2026-02-22T23:34:57.145 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29076.log 2026-02-22T23:34:57.145 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42026.log.gz 2026-02-22T23:34:57.145 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33046.log 2026-02-22T23:34:57.145 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30093.log: /var/log/ceph/ceph-client.admin.29076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30093.log.gz 2026-02-22T23:34:57.145 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29076.log.gz 2026-02-22T23:34:57.146 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41481.log 2026-02-22T23:34:57.146 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33046.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.33046.log.gz /var/log/ceph/ceph-client.admin.32301.log 2026-02-22T23:34:57.146 INFO:teuthology.orchestra.run.trial115.stderr: 2026-02-22T23:34:57.146 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32216.log 2026-02-22T23:34:57.146 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41481.log: /var/log/ceph/ceph-client.admin.32301.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41481.log.gz 2026-02-22T23:34:57.146 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32301.log.gz 2026-02-22T23:34:57.147 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32162.log 2026-02-22T23:34:57.147 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32216.log.gz 2026-02-22T23:34:57.147 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24749.log 2026-02-22T23:34:57.147 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42593.log 2026-02-22T23:34:57.147 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32162.log.gz 2026-02-22T23:34:57.148 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24749.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24749.log.gz 2026-02-22T23:34:57.148 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21292.log 2026-02-22T23:34:57.148 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32266.log 2026-02-22T23:34:57.148 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42593.log.gz 2026-02-22T23:34:57.148 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21292.log.gz 2026-02-22T23:34:57.148 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43246.log 2026-02-22T23:34:57.149 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32266.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32266.log.gz 2026-02-22T23:34:57.149 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29184.log 2026-02-22T23:34:57.149 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45882.log 2026-02-22T23:34:57.149 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43246.log: /var/log/ceph/ceph-client.admin.29184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43246.log.gz 2026-02-22T23:34:57.149 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.29184.log.gz -- 2026-02-22T23:34:57.149 INFO:teuthology.orchestra.run.trial115.stderr: /var/log/ceph/ceph-client.admin.21056.log 2026-02-22T23:34:57.149 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45882.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45882.log.gz 2026-02-22T23:34:57.150 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41814.log 2026-02-22T23:34:57.150 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.21056.log: -- /var/log/ceph/ceph-client.admin.36677.log 2026-02-22T23:34:57.150 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21056.log.gz 2026-02-22T23:34:57.150 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21870.log 2026-02-22T23:34:57.150 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41814.log.gz 2026-02-22T23:34:57.151 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45622.log 2026-02-22T23:34:57.151 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36677.log.gz 2026-02-22T23:34:57.151 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31860.log 2026-02-22T23:34:57.151 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21870.log.gz 2026-02-22T23:34:57.151 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45060.log 2026-02-22T23:34:57.151 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45622.log.gz 2026-02-22T23:34:57.152 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31860.log.gz 2026-02-22T23:34:57.152 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32604.log 2026-02-22T23:34:57.152 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45060.log.gz 2026-02-22T23:34:57.152 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33514.log 2026-02-22T23:34:57.152 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44017.log 2026-02-22T23:34:57.152 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32604.log.gz 2026-02-22T23:34:57.153 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33514.log.gz 2026-02-22T23:34:57.153 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22308.log 2026-02-22T23:34:57.153 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43481.log 2026-02-22T23:34:57.153 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44017.log.gz 2026-02-22T23:34:57.153 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22308.log.gz 2026-02-22T23:34:57.153 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29075.log 2026-02-22T23:34:57.154 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45334.log 2026-02-22T23:34:57.154 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43481.log.gz 2026-02-22T23:34:57.154 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45491.log 2026-02-22T23:34:57.154 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29075.log.gz 2026-02-22T23:34:57.154 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29995.log 2026-02-22T23:34:57.155 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45334.log.gz 2026-02-22T23:34:57.155 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45491.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45491.log.gz 2026-02-22T23:34:57.155 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21773.log 2026-02-22T23:34:57.155 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35500.log 2026-02-22T23:34:57.155 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29995.log.gz 2026-02-22T23:34:57.156 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27586.log 2026-02-22T23:34:57.156 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21773.log.gz 2026-02-22T23:34:57.156 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35500.log.gz 2026-02-22T23:34:57.156 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22361.log 2026-02-22T23:34:57.156 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44563.log 2026-02-22T23:34:57.156 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27586.log.gz 2026-02-22T23:34:57.157 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24742.log 2026-02-22T23:34:57.157 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22361.log.gz 2026-02-22T23:34:57.157 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44563.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44563.log.gz 2026-02-22T23:34:57.157 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35502.log 2026-02-22T23:34:57.157 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40802.log 2026-02-22T23:34:57.158 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24742.log: /var/log/ceph/ceph-client.admin.35502.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24742.log.gz 2026-02-22T23:34:57.158 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35502.log.gz 2026-02-22T23:34:57.158 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27621.log 2026-02-22T23:34:57.158 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.40802.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.40802.log.gz -- 2026-02-22T23:34:57.158 INFO:teuthology.orchestra.run.trial115.stderr: /var/log/ceph/ceph-client.admin.29855.log 2026-02-22T23:34:57.158 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32869.log 2026-02-22T23:34:57.159 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27621.log: /var/log/ceph/ceph-client.admin.29855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27621.log.gz 2026-02-22T23:34:57.159 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29855.log.gz 2026-02-22T23:34:57.159 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43480.log 2026-02-22T23:34:57.159 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32869.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32869.log.gz 2026-02-22T23:34:57.159 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42903.log 2026-02-22T23:34:57.159 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21136.log 2026-02-22T23:34:57.159 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43480.log.gz 2026-02-22T23:34:57.160 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43590.log 2026-02-22T23:34:57.160 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42903.log.gz 2026-02-22T23:34:57.160 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40793.log 2026-02-22T23:34:57.160 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21136.log.gz 2026-02-22T23:34:57.160 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43590.log.gz 2026-02-22T23:34:57.160 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40901.log 2026-02-22T23:34:57.161 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.40793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27716.log 2026-02-22T23:34:57.161 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40793.log.gz 2026-02-22T23:34:57.161 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.40901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40901.log.gz 2026-02-22T23:34:57.161 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24646.log 2026-02-22T23:34:57.161 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32500.log 2026-02-22T23:34:57.161 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27716.log.gz 2026-02-22T23:34:57.162 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31082.log 2026-02-22T23:34:57.162 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24646.log.gz 2026-02-22T23:34:57.162 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32500.log.gz 2026-02-22T23:34:57.162 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36669.log 2026-02-22T23:34:57.162 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31082.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.31082.log.gz 2026-02-22T23:34:57.162 INFO:teuthology.orchestra.run.trial115.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36847.log 2026-02-22T23:34:57.163 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21083.log 2026-02-22T23:34:57.163 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36669.log.gz 2026-02-22T23:34:57.163 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21110.log 2026-02-22T23:34:57.163 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36847.log.gz 2026-02-22T23:34:57.163 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21083.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21083.log.gz 2026-02-22T23:34:57.163 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36468.log 2026-02-22T23:34:57.164 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43783.log 2026-02-22T23:34:57.164 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21110.log.gz 2026-02-22T23:34:57.164 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-22T23:34:57.164 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36468.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36468.log.gz 2026-02-22T23:34:57.164 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43783.log.gz 2026-02-22T23:34:57.164 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36676.log 2026-02-22T23:34:57.165 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43820.log 2026-02-22T23:34:57.165 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.36676.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36676.log.gz 2026-02-22T23:34:57.165 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36145.log 2026-02-22T23:34:57.165 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28496.log 2026-02-22T23:34:57.166 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43820.log.gz 2026-02-22T23:34:57.166 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36145.log.gz 2026-02-22T23:34:57.166 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41480.log 2026-02-22T23:34:57.166 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.28496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30534.log 2026-02-22T23:34:57.166 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28496.log.gz 2026-02-22T23:34:57.167 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32640.log 2026-02-22T23:34:57.167 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41480.log.gz 2026-02-22T23:34:57.167 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.30534.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30534.log.gz 2026-02-22T23:34:57.167 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21162.log 2026-02-22T23:34:57.167 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43010.log 2026-02-22T23:34:57.167 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32640.log.gz 2026-02-22T23:34:57.168 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43695.log 2026-02-22T23:34:57.168 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21162.log.gz 2026-02-22T23:34:57.168 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43010.log.gz 2026-02-22T23:34:57.168 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17563.log 2026-02-22T23:34:57.168 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46143.log 2026-02-22T23:34:57.168 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43695.log.gz 2026-02-22T23:34:57.169 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.17563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36469.log 2026-02-22T23:34:57.169 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17563.log.gz 2026-02-22T23:34:57.169 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.46143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46143.log.gz 2026-02-22T23:34:57.169 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42229.log 2026-02-22T23:34:57.170 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.36469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43349.log 2026-02-22T23:34:57.170 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36469.log.gz 2026-02-22T23:34:57.170 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42229.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42229.log.gz 2026-02-22T23:34:57.170 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33240.log 2026-02-22T23:34:57.170 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35501.log 2026-02-22T23:34:57.170 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43349.log.gz 2026-02-22T23:34:57.171 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33240.log.gz 2026-02-22T23:34:57.171 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38664.log 2026-02-22T23:34:57.171 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35501.log.gz 2026-02-22T23:34:57.171 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22412.log 2026-02-22T23:34:57.171 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.38664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29752.log 2026-02-22T23:34:57.171 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38664.log.gz 2026-02-22T23:34:57.172 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43679.log 2026-02-22T23:34:57.172 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22412.log.gz 2026-02-22T23:34:57.172 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29752.log.gz 2026-02-22T23:34:57.172 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35599.log 2026-02-22T23:34:57.172 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21408.log 2026-02-22T23:34:57.172 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43679.log.gz 2026-02-22T23:34:57.172 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24442.log 2026-02-22T23:34:57.172 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35599.log.gz 2026-02-22T23:34:57.173 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21408.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21408.log.gz 2026-02-22T23:34:57.173 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33332.log 2026-02-22T23:34:57.173 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.24442.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24442.log.gz 2026-02-22T23:34:57.173 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.12504.log 2026-02-22T23:34:57.173 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31724.log 2026-02-22T23:34:57.173 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33332.log.gz 2026-02-22T23:34:57.173 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph.tmp-client.admin.12504.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.12504.log.gz 2026-02-22T23:34:57.173 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42561.log 2026-02-22T23:34:57.174 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35703.log 2026-02-22T23:34:57.174 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31724.log.gz 2026-02-22T23:34:57.174 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.42561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42561.log.gz 2026-02-22T23:34:57.174 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45830.log 2026-02-22T23:34:57.174 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.35703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21684.log 2026-02-22T23:34:57.174 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35703.log.gz 2026-02-22T23:34:57.175 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31965.log 2026-02-22T23:34:57.175 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45830.log.gz 2026-02-22T23:34:57.175 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45726.log 2026-02-22T23:34:57.175 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.21684.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21684.log.gz 2026-02-22T23:34:57.175 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43697.log 2026-02-22T23:34:57.175 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31965.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31965.log.gz 2026-02-22T23:34:57.175 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45726.log.gz 2026-02-22T23:34:57.176 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31954.log 2026-02-22T23:34:57.176 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.43697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33134.log 2026-02-22T23:34:57.176 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43697.log.gz 2026-02-22T23:34:57.176 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.31954.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31954.log.gz 2026-02-22T23:34:57.176 INFO:teuthology.orchestra.run.trial115.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.22510.log 2026-02-22T23:34:57.176 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.33134.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33134.log.gz 2026-02-22T23:34:57.176 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41038.log 2026-02-22T23:34:57.177 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.22510.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22510.log.gz 2026-02-22T23:34:57.177 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32734.log 2026-02-22T23:34:57.177 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41038.log.gz 2026-02-22T23:34:57.177 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27855.log 2026-02-22T23:34:57.177 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.32734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45700.log 2026-02-22T23:34:57.177 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32734.log.gz 2026-02-22T23:34:57.178 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27956.log 2026-02-22T23:34:57.178 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27855.log.gz 2026-02-22T23:34:57.178 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.45700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41343.log 2026-02-22T23:34:57.178 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45700.log.gz 2026-02-22T23:34:57.178 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.27956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27956.log.gz 2026-02-22T23:34:57.178 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44904.log 2026-02-22T23:34:57.179 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.41343.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41343.log.gz 2026-02-22T23:34:57.179 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29626.log 2026-02-22T23:34:57.179 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.44904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44904.log.gz 2026-02-22T23:34:57.179 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25106.log 2026-02-22T23:34:57.179 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.29626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29626.log.gz 2026-02-22T23:34:57.180 INFO:teuthology.orchestra.run.trial115.stderr:/var/log/ceph/ceph-client.admin.25106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25106.log.gz 2026-02-22T23:34:57.214 INFO:teuthology.orchestra.run.trial115.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2026-02-22T23:34:57.229 INFO:teuthology.orchestra.run.trial115.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-22T23:34:58.415 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2026-02-22T23:35:08.342 INFO:teuthology.orchestra.run.trial115.stderr: 94.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-22T23:35:11.518 INFO:teuthology.orchestra.run.trial115.stderr: 94.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-22T23:35:28.514 INFO:teuthology.orchestra.run.trial115.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-22T23:36:50.895 INFO:teuthology.orchestra.run.trial115.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2026-02-22T23:36:50.896 INFO:teuthology.orchestra.run.trial115.stderr: 2026-02-22T23:36:50.896 INFO:teuthology.orchestra.run.trial115.stderr:real 1m53.939s 2026-02-22T23:36:50.896 INFO:teuthology.orchestra.run.trial115.stderr:user 2m44.908s 2026-02-22T23:36:50.896 INFO:teuthology.orchestra.run.trial115.stderr:sys 0m6.931s 2026-02-22T23:36:50.898 INFO:tasks.ceph:Archiving logs... 2026-02-22T23:36:50.898 DEBUG:teuthology.misc:Transferring archived files from trial115:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64402/remote/trial115/log 2026-02-22T23:36:50.901 DEBUG:teuthology.orchestra.run.trial115:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-22T23:37:05.525 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-22T23:37:05.535 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-22T23:37:05.535 DEBUG:teuthology.orchestra.run.trial115:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-22T23:37:05.625 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-22T23:37:05.635 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-22T23:37:05.635 DEBUG:teuthology.orchestra.run.trial115:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T23:37:05.687 INFO:teuthology.orchestra.run.trial115.stdout: remote refid st t when poll reach delay offset jitter 2026-02-22T23:37:05.687 INFO:teuthology.orchestra.run.trial115.stdout:============================================================================== 2026-02-22T23:37:05.687 INFO:teuthology.orchestra.run.trial115.stdout:+rn-02.koehn.com 17.253.6.37 2 u 35 64 377 19.353 +0.081 3.406 2026-02-22T23:37:05.687 INFO:teuthology.orchestra.run.trial115.stdout:+172-104-28-175. 82.35.162.146 2 u 39 64 377 3.333 +2.521 2.443 2026-02-22T23:37:05.687 INFO:teuthology.orchestra.run.trial115.stdout:*pool-96-231-54- .E810. 1 u 36 64 377 11.615 -4.637 2.851 2026-02-22T23:37:05.687 INFO:teuthology.orchestra.run.trial115.stdout:-t2.time.bf1.yah 98.139.133.62 2 u 36 64 377 82.590 +35.259 2.880 2026-02-22T23:37:05.688 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-22T23:37:05.698 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-22T23:37:05.698 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-22T23:37:05.710 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-22T23:37:05.724 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-22T23:37:05.737 INFO:teuthology.task.internal:Duration was 778.537291 seconds 2026-02-22T23:37:05.737 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-22T23:37:05.751 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-22T23:37:05.751 DEBUG:teuthology.orchestra.run.trial115:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-22T23:37:05.774 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-22T23:37:05.774 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial115.front.sepia.ceph.com 2026-02-22T23:37:05.775 DEBUG:teuthology.orchestra.run.trial115:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-22T23:37:05.824 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-22T23:37:05.825 DEBUG:teuthology.orchestra.run.trial115:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T23:37:06.097 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-22T23:37:06.098 DEBUG:teuthology.orchestra.run.trial115:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T23:37:06.106 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T23:37:06.108 INFO:teuthology.orchestra.run.trial115.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T23:37:06.108 INFO:teuthology.orchestra.run.trial115.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T23:37:06.108 INFO:teuthology.orchestra.run.trial115.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-22T23:37:06.108 INFO:teuthology.orchestra.run.trial115.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-22T23:37:06.176 INFO:teuthology.orchestra.run.trial115.stderr: 81.9% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-22T23:37:06.178 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-22T23:37:06.196 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-22T23:37:06.196 DEBUG:teuthology.orchestra.run.trial115:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-22T23:37:06.227 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-22T23:37:06.243 DEBUG:teuthology.orchestra.run.trial115:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-22T23:37:06.278 INFO:teuthology.orchestra.run.trial115.stdout:kernel.core_pattern = core 2026-02-22T23:37:06.288 DEBUG:teuthology.orchestra.run.trial115:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-22T23:37:06.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T23:37:06.334 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-22T23:37:06.350 INFO:teuthology.task.internal:Transferring archived files... 2026-02-22T23:37:06.351 DEBUG:teuthology.misc:Transferring archived files from trial115:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64402/remote/trial115 2026-02-22T23:37:06.352 DEBUG:teuthology.orchestra.run.trial115:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-22T23:37:06.415 INFO:teuthology.task.internal:Removing archive directory... 2026-02-22T23:37:06.415 DEBUG:teuthology.orchestra.run.trial115:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-22T23:37:06.443 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-22T23:37:06.459 INFO:teuthology.task.internal:Not uploading archives. 2026-02-22T23:37:06.459 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-22T23:37:06.475 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-22T23:37:06.475 DEBUG:teuthology.orchestra.run.trial115:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-22T23:37:06.490 INFO:teuthology.orchestra.run.trial115.stdout: 1179687 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 22 23:37 /home/ubuntu/cephtest 2026-02-22T23:37:06.492 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-22T23:37:06.509 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-22T23:37:06.545 INFO:teuthology.run:Summary data: description: rados/singleton/{all/ec-inconsistent-hinfo mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} duration: 778.5372912883759 flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com success: true 2026-02-22T23:37:06.546 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-22T23:37:06.671 INFO:teuthology.run:pass